How should we measure school performance and hold schools accountable? A study of competing statistical methods and how they compare to Progress 8

Lead Research Organisation: University of Bristol
Department Name: Education

Abstract

In 2016, the Department for Education radically overhauled their secondary school accountability system and introduced 'Progress 8', arguing it to be the simplest and fairest school performance measure to date. Progress 8 aims to quantify and communicate the average academic value each school adds to their pupils' learning. Specifically, Progress 8 measures how much higher each school's pupils score in their age 16 GCSE examinations than expected given their age 11 KS2 test scores when they started secondary schooling. Progress 8 scores are used to hold schools to account, with the lowest scoring schools judged 'underperforming' and 'coasting'; classifications that trigger intense scrutiny and intervention from the school inspectorate, Ofsted. Given the high-stakes involved, research is urgently needed to first evaluate the strengths and weaknesses of the statistical method underlying Progress 8, and second, to explore the potential benefits of alternative methods for measuring school performance. Our proposed research will address these needs and in doing so will advance scientific understanding about school performance measurement.

The first aim of the proposed research is to study how sensitive Progress 8 and other value-added measures' scores, rankings and classifications are to the statistical modelling decisions and assumptions implicit in their design. These include fundamental decisions over whether value-added measures should additionally consider pupil non-academic outcomes and whether they should account for the substantial differences in pupils' demographic and socioeconomic characteristics between schools. We will also explore an assumption implicit in Progress 8 and many other value-added measures which is that there is no relationship between pupils' family backgrounds and the quality of the schools they attend (i.e., confounding or selection into school). We will study the extent to which alternative value-added approaches can address this (using fixed- and random-effects models). We will also explore another assumption of Progress 8 and many other value-added measures which is that the GCSE outcomes of low scoring pupils at KS2 are no more variable than those of high scoring pupils at KS2 (i.e., homoskedasticity). We will study different approaches proposed in the literature to address this concern.

The second aim of the proposed research is to adapt and evaluate a range of cutting edge statistical methods not yet applied to measuring school performance. We will study their advantages and disadvantages over standard value-added approaches including Progress 8. These include methods that allow users to compare not just the average progress made by pupils in each school, but to also study the variability in pupils' progress (i.e., mixed-effects location scale models and multilevel quantile regression). These also include methods that are less reliant on extrapolating the relationships in the data and may therefore lead to fairer and more meaningful school comparisons (i.e., propensity score and other matching methods for comparing multiple treatments).

Whereas the above aims will advance the academic research in the school effectiveness and methods literatures, our third aim is to make fundamental contributions to the wider non-academic understanding of alternative methods for measuring school performance in England. We will therefore disseminate our findings to the users and producers of school and public-sector institution performance measures. These groups include policy makers and their advisers, statisticians and researchers in relevant Government departments and charities, as well as schools, the media and parents. Planned activities include: knowledge exchange meetings, research briefings, a data visualization website, an end-of-grant symposium, online training materials, and a face-to-face short training course.

Planned Impact

WHO WILL BENEFIT FROM THIS RESEARCH?

GROUP A are the USERS OF PROGRESS 8 who need to be able to understand the strengths and weaknesses of the published statistics. This group includes: Department for Education (DfE) policymakers and their advisors who use Progress 8 to classify schools as 'underperforming', 'coasting' or 'high progress'; Ofsted who use these data to target and inform their school inspections; third sector organisations, such as the Education Endowment Foundation, who use Progress 8 to select schools for their randomised control trials and to assess the effectiveness of their interventions; local authorities, academy chains and schools, who use Progress 8 for performance monitoring and resource allocation; the media, who present and explain Progress 8 scores to the public; and parents who are encouraged to use Progress 8 to choose which schools to send their children to.

GROUP B are the PRODUCERS OF SCHOOL AND OTHER PUBLIC-SECTOR INSTITUTION PERFORMANCE MEASURES who need to understand the different statistical methods, choices and issues involved when designing indicators. This group includes: DfE statisticians responsible for Progress 8 and the Fischer Family Trust and other commercial and charitable organisations who produce alternative school performance measures to support target setting and self-evaluation. Internationally, this group includes an increasing number of school systems using pupil test score data to hold schools to account. In terms of other public-sector areas, this group includes: the Higher Education Funding Council for England who are developing university performance measures of 'learning gain', potentially to be included as part of the Teaching Excellence Framework; the Department of Health who publish case-mix adjusted hospital mortality rates; and the Home Office who predict context adjusted police force crime rates.


HOW WILL THEY BENEFIT FROM THIS RESEARCH?

GROUP A will benefit from our substantive research which will carefully explain what Progress 8 attempts to do, how and why it ignores pupil demographic and socioeconomic characteristics and the implications this has for making fair comparisons across schools; and the influence this and other decisions surrounding the design and presentation of Progress 8 have on schools' scores, ranking and classifications. This group will also benefit from our investigations into the risk of over interpreting small differences in schools' performances, and how this will likely prove particularly important when comparing within school socioeconomic and other Progress 8 performance gaps. The novel statistical methods we shall explore will generate new substantive insights into the effectiveness of schools in England and this will further benefit this group and feed directly into national debates and policy surrounding school improvement and equity as well as school accountability.

GROUP B will benefit from our methodological research which explores the potential statistical strengths and weakness of not only different value-added approaches, but also of a range of cutting edge methods not yet applied to measuring school performance. These include mixed-effects location scale models, multilevel quantile regression and matching methods. This will enable this group to make more informed choices when developing their own measures. This group will also benefit from the training materials we shall develop as well as any versions of these adapted by others to their disciplines. The materials will provide step-by-step instructions on how to implement the different statistical methods in standard software. They will also act as exemplars as to how to richly interpret the estimated institution effects generated by the different methods as well as good practices in communicating institution performances and their uncertainty to lay audiences.


WHAT WILL BE DONE TO ENSURE THAT THEY BENEFIT FROM THE RESEARCH?

See PATHWAYS TO IMPACT.

Publications

10 25 50
 
Description This grant is funding three years of intensive research on England's Government school performance tables. The grant is focusing on statistically critiquing and comparing various school performance measures published in these tables, especially value-added and progress measures. We are particularly focussing on the influence of choice of statistical modelling and estimation approach on the resulting performance measures.


First paper published:

Leckie, G. and Goldstein, H. (2019) The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England. British Educational Research Journal. DOI: 10.1002/berj.3511.

Abstract:
In the UK, USA and elsewhere, school accountability systems increasingly compare schools using value-added measures of school performance derived from pupil scores in high-stakes standardised tests. Rather than naïvely comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils make during a year or phase of schooling. Schools, however, also differ in terms of their pupil demographic and socioeconomic characteristics and these factors also predict why some schools subsequently score higher than others. Many therefore argue that value-added measures unadjusted for pupil background are biased in favour of schools with more 'educationally advantaged' intakes. But others worry that adjusting for pupil background entrenches socioeconomic inequities and excuses low-performing schools. In this article we explore these theoretical arguments and their practical importance in the context of the 'Progress 8' secondary school accountability system in England, which has chosen to ignore pupil background. We reveal how the reported low or high performance of many schools changes dramatically once adjustments are made for pupil background, and these changes also affect the reported differential performances of regions and of different school types. We conclude that accountability systems which choose to ignore pupil background are likely to reward and punish the wrong schools and this will likely have detrimental effects on pupil learning. These findings, especially when coupled with more general concerns surrounding high-stakes testing and school value-added models, raise serious doubts about their use in school accountability systems.


Second paper published:

Leckie, G., Browne, W., Goldstein, H., Merlo, J., Austin, P. Variance partitioning in multilevel models for count data. Psychological Methods. Forthcoming.

Abstract:
A first step when fitting multilevel models to continuous responses is to explore the degree of clustering in the data. Researchers fit variance-component models and then report the proportion of variation in the response that is due to systematic differences between clusters. Equally they report the response correlation between units within a cluster. These statistics are popularly referred to as variance partition coefficients (VPCs) and intraclass correlation coefficients (ICCs). When fitting multilevel models to categorical (binary, ordinal, or nominal) and count responses, these statistics prove more challenging to calculate. For categorical response models, researchers appeal to their latent response formulations and report VPCs/ICCs in terms of latent continuous responses envisaged to underly the observed categorical responses. For standard count response models, however, there are no corresponding latent response formulations. More generally, there is a paucity of guidance on how to partition the variation. As a result, applied researchers are likely to avoid or inadequately report and discuss the substantive importance of clustering and cluster effects in their studies. A recent article drew attention to a little-known exact algebraic expression for the VPC/ICC for the special case of the two-level random-intercept Poisson model. In this article, we make a substantial new contribution. First, we derive exact VPC/ICC expressions for more flexible negative binomial models that allows for overdispersion, a phenomenon which often occurs in practice. Then we derive exact VPC/ICC expressions for three-level and random-coefficient extensions to these models. We illustrate our work with an application to student absenteeism.


First paper under review:

Leckie, G., Prior, L., Goldstein, H. (2019). The implications of Labour's plan to scrap Key Stage 2 tests for Progress 8 and secondary school accountability in England. arXiv: 1911.06884 [stat AP].

In England, Progress 8 is the Conservative government's headline secondary school performance and accountability measure. Progress 8 attempts to measure the average academic progress pupils make in each school between their KS2 tests and their GCSE Attainment 8 examinations. The Labour opposition recently announced they would scrap the KS2 tests were they to be elected. Such a move, however, would preclude the publication of Progress 8 and would leave schools to be compared in terms of their average Attainment 8 scores or, at best, their Attainment 8 scores only adjusted for school differences in pupil demographic and socioeconomic characteristics. In this paper, we argue and illustrate empirically that this best-case scenario of an 'Adjusted Attainment 8' measure would prove less fair and meaningful than Progress 8 and therefore a backwards step, especially when Progress 8 itself has been criticised as biased against schools teaching educationally disadvantaged intakes.


Second paper under review:

Prior, L., Goldstein, H., & Leckie, G. (2020). School value-added models for multivariate academic and non-academic outcomes: A more rounded approach to using student data to inform school accountability. arXiv:2001.01996 [stat AP].

Education systems around the world increasingly rely on school value-added models to hold schools to account. These models typically focus on a limited number of academic outcomes, failing to recognise the broader range of non-academic student outcomes, attitudes and behaviours to which schools contribute. We explore how the traditional multilevel modelling approach to school value-added models can be extended to simultaneously analyse multiple academic and non-academic outcomes and thereby can potentially provide a more rounded approach to using student data to inform school accountability. We jointly model student attainment, absence and exclusion data for schools in England. We find different results across the three outcomes, in terms of the size and consistency of school effects, and the importance of adjusting for student and school characteristics. The results suggest the three outcomes are capturing fundamentally distinct aspects of school performance, recommending the consideration of non-academic outcomes in systems of school accountability.


Working paper:

Goldstein, H., Leckie, G., Prior, L. (2020). Providing educational accountability for Local Authorities based upon sampling pupils within schools: moving away from simplistic school league tables. arXiv:2002.09897 [stat AP].

This paper proposes an alternative educational accountability system for England that moves away from simplistic comparisons, or league tables, among schools towards a more nuanced reporting at a level of education authorities or other school groupings. Based upon the sampling of pupils within schools, it proposes the use of quantitative pupil assessment data as screening devices that provide evidence within an integrated accountability system that includes inspection. At the same time, it aims to provide a richer set of data than currently available for research as well as accountability purposes. We argue that if carefully implemented within a context of school improvement, such a system has the potential to largely eliminate the deleterious side effects and curriculum distortions of the present system. While being proposed within the context of the current English secondary school system, the proposals will have relevance for other phases of schooling and similar systems in other countries.
Exploitation Route Our findings might be used by Government (Department for Education) in how they better design future school performance measures as well as the increasingly large number of charity and commercial companies who use the published school performance measures in performance monitoring systems which they sell to schools (e.g., Fischer Family Trust). Ofsted could change the way they view and use DfE data when inspecting schools.
Sectors Education

URL http://www.bristol.ac.uk/cmm/research/school-performance/
 
Description Policy context Progress 8 is the Department for Education's (DfE) headline school performance and accountability measure for all state secondary schools in England and is published annually in their school league tables (see https://www.gov.uk/school-performance-tables). Progress 8 aims to measure the average academic "progress" pupils make over secondary schooling by comparing their end of secondary schooling GCSE exam results to their end of primary schooling KS2 test results. The DfE and Ofsted both rely heavily on Progress 8 to hold schools to account and so schools' futures are very much dictated by this measure. Our research Our research shows that Progress 8 is an unfair measure of school performance as while it accounts for school intake differences in pupils' KS2 results, it still ignores school intake differences in all other pupil background characteristics, yet these also predict why some schools score higher at GCSE than others (Leckie and Goldstein, 2019). This work is part of our broader three-year Economic and Social Research Council (ESRC) grant funded project on the school performance tables (see http://www.bristol.ac.uk/cmm/research/school-performance/). Our statistical analyses of the DfE's own data shows that adjusting for pupil background would see the national Progress 8 school league table rankings of one-fifth of schools change by over 500 places. Furthermore, 40% of schools judged 'underperforming' would move up out of this banding. The main driver of these results is that Progress 8 penalises schools teaching above average proportions of free school meal and white British pupils, two pupil groups which struggle nationally. We conclude that Progress 8 is biased against schools with socially disadvantaged pupil groups and therefore punishes and rewards the wrong schools. We recommend the DfE adjust Progress 8 for pupil backgrounds to provide fairer and more meaningful summaries of the impact that schools have on pupil learning. More broadly, however, we argue far less emphasis be placed on all school performance measures as a failure to adjust for pupil background is just one of many statistical and more general concerns that we and others have raised with current high-stakes testing and league tables in England (Leckie and Goldstein, 2017). Our impact Our research has had impact from the start, via two-way knowledge exchange meetings with multiple stakeholders. First, we presented to local schools in Bristol. Then we shared our initial findings with Ofsted at their Bristol offices who prompted us to conduct useful further analyses. We then shared our draft research with the Fischer Family Trust, an organisation which sells school performance monitoring tools to schools, and used their feedback to further improve our work. A second meeting with Ofsted helped further refine our analyses. Next, we discussed the relevance of our findings for OfS and their related work on measuring University 'learning gains'. Most recently, we had a rich and varied discussion with DfE policy advisors in their Westminster offices around the likely future direction of school performance measures in England. The University of Bristol press release of our work led to national radio (Talk Sport and Heart FM), newspaper and online coverage (The Guardian, The Independent, The Times, and Times Educational Supplement) leading to increased public awareness and understanding or our work. Following this media attention, we gave an invited talk to teachers at the National Education Union and have presented our work at an invited private meeting with the Education Select Committee. We have since had a follow-on face-to-face meeting with Emma Hardy, Labour MP (now Shadow FE and HE Minister). This meeting included discussing alternatives to Progress 8 and data-driven school accountability given Labour's May 2019 announcement to scrap KS2 tests. Our work has also prompted the Co-Op Academies Trust to issue their own press release citing our work and calling for Progress 8 to be adjusted and encouraging other schools to contact us. This lead to more media attention this time in TES and Schools Week. Following this, we have corresponded with many headteachers who have emailed to find out their pupil background adjusted Progress 8 scores and to ask to be involved in future research. Frank Norris, Director of Co-Op Academies Trust, has also discussed our work in a meeting with Labour MP Angela Rayner (Shadow Secretary of State for Education) and 25 Labour MPs. We will soon be presenting to all the Co-Op headteachers in Manchester and the Frank Norris will be visiting us in Bristol. We have also collaborated extensively with The Northern Powerhouse Partnership (NPP) who have championed our work at a second Education Select Committee on 'Education in the North' and in a meeting with shadow cabinet members on education and skills. Partnering with the NPP, we have then published Adjusted Progress 8 scores for all schools in England and on a dedicated interactive website (see http://www.northernpowerhousepartnership.co.uk/publications/latest-research-shows-league-tables-punish-and-reward-wrong-schools/) as a way of encouraging further debate around concerns with Progress 8 and data driven accountability and to place more pressure on the DfE to engage on these issues. Lucy Powell, Labour MP, Education Select Committee member and former Shadow Secretary of State for Education said in her foreword to this work "This is a ground-breaking piece of work by the Norther Powerhouse Partnership with Bristol University". The press release of this joint work led to further media coverage by the BBC, Independent, FT, and Times Education Supplement. References • Leckie, G., & Goldstein, H. (2017). The evolution of school league tables in England 1992-2016: 'contextual value-added', 'expected progress' and 'progress 8'. British Educational Research Journal, 43(2), 193-212. URL: https://onlinelibrary.wiley.com/doi/full/10.1002/berj.3264 • Leckie, G. and Goldstein, H. (2019). The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England. British Educational Research Journal, 45(3), 518-537. URL: https://onlinelibrary.wiley.com/doi/10.1002/berj.3511 • Leckie, G., Prior, L., & Goldstein, H. (2019). The implications of Labour's plan to scrap Key Stage 2 tests for Progress 8 and secondary school accountability in England. arXiv:1911.06884 [stat.AP]. URL: https://arxiv.org/abs/1911.06884
First Year Of Impact 2018
Sector Education
Impact Types Policy & public services

 
Description Discussions with Bulgarian minister of education and heads of directorates as expert member of World Bank team
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Israel Academy of Sciences: Discussion about school value-added models, 2019
Geographic Reach Asia 
Policy Influence Type Participation in a advisory committee
 
Description Knowledge exchange meeting - Fischer Family Trust. Discussed Progress 8 review paper. December 2020.
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Knowledge exchange meeting - Fischer Family Trust. Discussed their related research work. January 2021.
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Knowledge exchange meeting. August 2019. Fischer Family Trust. Discussed their related work.
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Knowledge exchange meeting. December 2018. Fischer Family Trust. Discussed adjusted Progress 8 work.
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Knowledge exchange meeting. July 2020. Discussion with Director and Chief Economist of Northern Powerhouse Partnership
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Knowledge exchange meeting. May 2019. Discussion with Director and Chief Economist of Northern Powerhouse Partnership
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Office for Standards in Education, Children's Services and Skills (Ofsted) knowledge exchange meeting about potential Ofsted projects and Progress 8 research
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Office for Standards in Education, Children's Services and Skills (Ofsted) knowledge exchange meeting about potential Ofsted projects and Progress 8 research
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Office for Standards in Education, Children's Services and Skills (Ofsted): knowledge exchange meeting for Ofsted about general research collaboration with University of Bristol and potential PhD studentship
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Office for Student (OfS) Benchmarking review advisory group
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Office for Students (OfS), Benchmarking review advisory group
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Presentation of Progress 8 research to Emma Hardy MP (Education Select Committee Member) at Portcullis House
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Presentation of our Progress 8 research at UK Parliament Education Select Committee
Geographic Reach National 
Policy Influence Type Gave evidence to a government review
 
Description Submitted statement to Ofqual consultation "GCSE and A level grading proposals for 2020"
Geographic Reach National 
Policy Influence Type Participation in a national consultation
 
Description Submitted written evidence to Education Select Committee on "Left behind white pupils from disadvantaged backgrounds"
Geographic Reach National 
Policy Influence Type Gave evidence to a government review
 
Description Sweden Department for Education: Advising on school value-added models
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description Trinidad and Tobago Department for Education: Advising on school value-added models
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description World Bank Expert advisor on school value-added modelling project
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
 
Description 'Progress 8' National Education Union annual conference, Liverpool 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact April 2019
Year(s) Of Engagement Activity 2019
 
Description 'School accountability in crisis - holding the judges to accounts', British Educational Research Association (BERA) 2019 Annual Conference Joint coordinator with Terry Wrigley and Tom Perry 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact September 2019
Year(s) Of Engagement Activity 2019
 
Description 'School value-added models of academic and non-academic student outcomes: A more rounded approach to using student data to inform school accountability with application to attainment, absences and exclusions in England. NPD User Group Meeting, London. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact January 2020
Year(s) Of Engagement Activity 2020
 
Description 'Secondary school performance measures and league tables in England: An overview of the official measures and their statistical modelling, data, policy, and interpretation issues', Statistics and Probability Group, University of York 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact February 2020
Year(s) Of Engagement Activity 2020
 
Description 'Should the Government modify Progress 8 to account for school intake differences in pupil background?' Afternoon event on 'Contextual Data: The benefits and the pitfalls for schools and society', Co-Op and Northern Powerhouse Partnership, Manchester 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact March 2020
Year(s) Of Engagement Activity 2020
 
Description 'Should the Government modify Progress 8 to account for school intake differences in pupil background?' One day conference on 'Redefining progress for disadvantaged students'. Education Observatory, University of Wolverhampton 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact January 2020
Year(s) Of Engagement Activity 2020
 
Description A talk of presentaton: 'Accountability and school differential effects' Department of Learning and Leadership, IOE UCL, London. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Schools
Results and Impact November 2018
Year(s) Of Engagement Activity 2018
URL https://mediacentral.ucl.ac.uk/Play/15633
 
Description A talk or presentation: 2018 AERA Annual Meeting, New York 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Apri 2018
Year(s) Of Engagement Activity 2018
 
Description A talk or presentation: Bristol Cathedral School, Bristol 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact September 2018
Year(s) Of Engagement Activity 2018
 
Description A talk or presentation: Educational Effectiveness EARLI SIG 18 & 23 meeting. Groningen: 25 years of school league tables, accountability and choice: Lessons from England 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact August 2018
Year(s) Of Engagement Activity 2018
 
Description A talk or presentation: Q-step Seminar Series, University of Exeter 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact May 2018
Year(s) Of Engagement Activity 2018
 
Description A talk or presentation: Royal Statistical Society Annual Conference, Cardiff 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact September 2018
Year(s) Of Engagement Activity 2018
 
Description BERA Podcast recorded seminar. 'The importance of adjusting for pupil sociodemographics in Progress 8 and school accountability in England' 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Recorded seminar for BERA podcast on adjusting for pupil background characteristics in Progress 8 school performance measure.
Year(s) Of Engagement Activity 2020
 
Description Blog post: Leckie G, Prior L, Goldstein H (2019) School accountability by Progress 8: A critique of Conservative and Labour proposals 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact Blog post written for BERA blog in run-up to 2019 General Election, focusing on potential implications for school accountability of Conservative and Labour standpoints. Intended to spark discussion.
Year(s) Of Engagement Activity 2019
URL https://www.bera.ac.uk/blog/school-accountability-by-progress-8-a-critique-of-conservative-and-labou...
 
Description Blog post: Leckie G, Prior L, Goldstein H (2020). School accountability by Progress 8: A critique of Conservative and Labour proposals. School of Education Blog, University of Bristol. 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact Blog post written for the School of Education focused on school accountability in reference to proposals and standpoints of the Conservative and Labour parties. Intended to spark discussion.
Year(s) Of Engagement Activity 2019
URL https://schoolofeducation.blogs.bristol.ac.uk/2020/01/23/school-accountability-by-progress-8-a-criti...
 
Description Calculating intraclass correlation coefficients in multilevel models for count responses. 12th International Amsterdam Multilevel Conference, Utrecht, April 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact A presentation to an international audience of academics and postgraduate researchers interested in use of multilevel models.
Year(s) Of Engagement Activity 2019
 
Description Catherine Peckham Symposium/Otto Wolff Lecture 2019, London 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact October 2019.
Year(s) Of Engagement Activity 2019
 
Description DfE funded NPD User Group meeting (1 day), Bristol. Co-organiser with Rebecca Allen 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Third sector organisations
Results and Impact September 2018
Year(s) Of Engagement Activity 2018
 
Description Discussion with Director and Chief Economist of Northern Powerhouse Partnership 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Policymakers/politicians
Results and Impact 2019
Year(s) Of Engagement Activity 2019
 
Description Discussion with Education Campaigns Manager at Humanists UK about Progress 8 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Third sector organisations
Results and Impact August 2019
Year(s) Of Engagement Activity 2019
 
Description Discussion with Head of Co-Op Academies about Progress 8 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact February 2019
Year(s) Of Engagement Activity 2019
 
Description Discussion with Head of Co-Op Academies about Progress 8 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact April 2019
Year(s) Of Engagement Activity 2019
 
Description Discussion with Headmaster of school in North West about Progress 8 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Schools
Results and Impact October 2018
Year(s) Of Engagement Activity 2018
 
Description Leckie, G., Prior, L., Goldstein, H. (2019). School accountability by Progrress 8: A critique of Conservative and Labour proposals. BERA Election Blog post. 29/11/2019 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Blog post considering the potential implications of policy proposals by the Conservatives and Labour on the Progress 8 school performance measure.
Year(s) Of Engagement Activity 2019
 
Description Leckie, G., and Goldstein, H. (2020). Progress 8 school league tables punish and reward the wrong schools. BERA's Research Intelligence magazine, 143. 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Article for magazine on Progress 8 and the implications of adjustment for student background characteristics.
Year(s) Of Engagement Activity 2020
 
Description Newspaper coverage: Bristol Post, Daily Express, The Guardian, The Independent, Leigh Journal, The Star, The Times, Times Educational Supplement, Warrington Guardian, Wigan Today, Yorkshire Post 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact January 2019
Year(s) Of Engagement Activity 2019
 
Description November 2020: Frank Norris: "Why the Co-op is involved in the academy programme". Bristol Conversations in Education. School of Education, University of Bristol. Organiser G. Leckie. 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Postgraduate students
Results and Impact Organised talk by Frank Norris on Co-op's involvement in the academies programme for the Bristol Conversations in Education series.
Year(s) Of Engagement Activity 2020
 
Description Other media coverage: BBC website, Chronical Live, FE News 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact 2019
Year(s) Of Engagement Activity 2019
 
Description Other media coverage: Ekklesia, Greensheets, Humanists UK, Local Gov, My Science, Reclaiming Schools, Schools Improvement, Schools Week, World News Live 4 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact January 2019
Year(s) Of Engagement Activity 2019
 
Description Policy Briefing: Leckie, G., Goldstein, H. (2019). The Government's Progress 8 school performance measure needs to account for pupil background. PolicyBristol. Policy Briefing 66: 01/2019. 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact January 2019
Year(s) Of Engagement Activity 2019
 
Description Postgraduate short course: Introduction to Multilevel Modelling (1 day). Bamberg. Course Instructor 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact October 2018
Year(s) Of Engagement Activity 2018
 
Description Postgraduate short course: Multilevel Modelling in School Effectiveness Research (0.5 days). EARLI Special Interest Group 18 Educational Effectiveness, Groningen. Course Instructor 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact August 2018
Year(s) Of Engagement Activity 2018
 
Description Radio coverage: Heart radio (pre-recorded interview); Love sport radio (live interview); BBC World at One (invited to live interview, declined as I was abroad). 
Form Of Engagement Activity A press release, press conference or response to a media enquiry/interview
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Media (as a channel to the public)
Results and Impact January 2019
Year(s) Of Engagement Activity 2019
 
Description Roundtable discussion on Progress 8 and school accountability measures with 15 others Held online. 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact A roundtable discussion about Progress 8 and school accountability measures with academics, and others from the Fischer Family Trust, Education Policy Institute, Centre for Education and Youth, and the Behavioural Insights Team, as well as representatives from the DfE.
Year(s) Of Engagement Activity 2020
 
Description Video recorded seminar: 'Accountability and school differential effects', Department of Learning and Leadership, IOE UCL, London. Speaker 
Form Of Engagement Activity A broadcast e.g. TV/radio/film/podcast (other than news/press)
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact 2018
Year(s) Of Engagement Activity 2018
 
Description Website: Fairer Schools Index 2018. Northern Powerhouse Partnership collaboration. 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact In joint work with the Northern Powerhouse Partnership we produced a website where schools could look up their Progress 8 and Adjusted Progress 8 scores. This sparked media interest and enquiries from those in the schools sector about scores and the change under adjustment for pupil background characteristics.
Year(s) Of Engagement Activity 2018
URL http://orthernpowerhousepartnership.co.uk/publications/latest-research-shows-league-tables-punish-an...
 
Description Website: Fairer Schools Index 2019. Northern Powerhouse Partnership collaboration. 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Schools
Results and Impact In joint work with the Northern Powerhouse Partnership we produced a website where schools could look up their Progress 8 and Adjusted Progress 8 scores. This was accompanied by media interest and sparked further interest from those involved in the schools sector.
Year(s) Of Engagement Activity 2019
URL https://www.arcgis.com/apps/dashboards/1741a670cfcb493eb2cb20f14af8a064