The kinds of poverty in schools and their impact on student progress

Lead Research Organisation: Durham University
Department Name: Education

Abstract

There is already considerable evidence that the nature of school intakes matters. Schools vary in terms of the proportion of pupils with learning challenges or from disadvantaged backgrounds, for example. And this affects the resources required by each school, the progress that can be expected in terms of pupil attainment, and even pupils' sense of society and what would be an appropriate future for them. This new research is based on consideration of a newly constructed indicator of disadvantage at school, using it for a new approach to make considerations of school intakes both more accurate and fairer for all concerned.

The research questions are:

1) What is the pattern of pupil FSM 'trajectories' over their school careers? What are the characteristics of pupils who enter and leave FSM eligibility compared to those who remain eligible throughout their school life, those who have never been eligible, and those who are missing FSM data at some stage?
2) What difference does including the pupil FSM trajectories make to the apparent patterns of segregation between school intakes? Are permanently eligible or missing-FSM pupils clustered in particular areas, types of schools or schools?
3) What are the patterns of attainment of pupils with different FSM trajectories? What are the implications of this analysis for instruments of policy such as the Pupil Premium gap?
4) What difference does including the FSM trajectories make to school average progress scores? Would instruments of school performance policy, such as progress scores, be fairer given more detail about the more precise level of relative disadvantage in school intakes?

The results will be relevant for policy and practice in England, especially concerning attempts to reduce or overcome the poverty gradient in school attainment and subsequent educational participation. This study also contributes to important theoretical issues, and creates new knowledge about school systems and their performance. The study will use ESRC-supported existing official datasets to describe, better than in previous accounts, the changing nature of intakes to schools, and the link between pupil intake characteristics and subsequent attainment/participation. Previous studies and official accounts use FSM-eligibility as a binary (Y/N) variable, as opposed to the finer grained
variable to be used for these new calculations. The focus will be on secondary state-funded mainstream schools in the National Pupil Database, following cohorts from 2003/4 onwards from their primary schools to Year 11, looking at pupils moving in and out of eligibility for free school meals (FSM), and those missing relevant data. It will collate the other characteristics and the outcomes for these different patterns of pupil FSM 'trajectories', and recalculate existing and published indices of FSM segregration between schools, school types, and regions. It will use all of these results to create statistical models to assess the extent to which consideration of these different categories of pupils living in poverty - such as never eligible, recently became FSM, previously FSM, moving in and out of FSM-eligibility, and missing relevant data - changes our understanding of how serious FSM segregation between schools is, what impacts it has, what causes it, and how it relates to attainment and pupil progress.

Instruments of school performance policy, such as progress scores, may not be fair in the absence of more detail about the more precise level of relative disadvantage in school intakes, as will be provided by this study and its consideration of 'trajectories' of FSM-eligiblity over time and their interaction with other pupil background characteristics. Also policies such as the pupil premium can be made more efficient and effective, by directing extra funding to where it is most needed. The project will contribute to a better education system, a more efficient use of taxpayer-funding and a fairer society

Planned Impact

The study has the potential to influence government policy, international and local policy-makers, stakeholders such schools and teachers, the context for school inspections, and the work of those attempting to overcome disadvantage in education. Whatever the results are they will be relevant to judgements about school performance, the fairness of school place allocation procedures, inspection findings by providing context for the pupil premium gap in each school, and teacher and departmental effectiveness judgements. They will also provide a focus for where interventions to overcome the effects of poverty on schooling would be most needed or most effective

Poverty, as assessed by FSM-eligibility, is routinely used as context for judging both individual- and school-level attainment, as an indicator of school composition, and as the basis for the pupil premium (PP) funding policy. PP itself is
important for current policies based on assessing the pupil premium gap in schools, including the work of OFSTED, RAISE, the National pupil premium Champion, and various school awards. As shown by the preparatory work for this bid, many of the calculations underlying such policies/practices may be unwittingly misleading, and unfair to certain types of schools and regions. Knowledge of the quality, reach and limitations of FSM as an indicator is therefore fundamental to accurate decision-making in all of these important areas. The results should lead to improved understanding of the nature and impact of school intakes, and understanding the impact of peers and school intake clustering on attainment and progress. The findings will have considerable implications for the use of evidence from the Pupil Premium Toolkit, and the ways in which school performance is analysed more generally. A wider community, including admission authorities, individual schools and families will also be interested, as will those concerned with school choice policies around the world.

An important part of this project will be to engineer the evidence found into an easily usable form for policy-makers and practitioners. These forms will include evidence summaries, press releases, and evidence-based recommendations. We will use a variety of our regular channels for disseminating the results in usable formats - including broadcast and print media, House of Commons Select Committee, think-tanks, admissions authorities and active groups such as the Campaign for State Education.

Given the ambitions of the project it is important to engage intended beneficiaries of the research from the outset, and several key organisations including the Joseph Rowntree Foundation, the Educational Endowment Foundation, the ATL,, and seven local authorities in areas of high disadvantage have agreed to be part of our user group already. They will help shape the objectives and progress of the research, and to create outcomes in a form that will be most useful for policy-makers and practitioners. We anticipate them being contributors from the outset, and major channels for disseminating the findings and recommendations from our research (see Letters of Support). We will add further users and groups as the study progresses.

Once this work has been completed, it should also prepare the way for further primary research in the kinds of settings represented by the local authorities in the user groups, pursuing the results of the secondary analyses into field settings.

See also Academic Beneficiaries for publication and other dissemination plans.

Publications

10 25 50
 
Description Generic conclusions about the research:The ECR who was CI on the grant now has a permanent academic post, and is PI on her own application for ESRC funding, having had training, presented to academics and the public, and published in the course of the project.

The project end date is over, and extended by six moths, and ADRN has still not been able to deliver on an agreed data request submitted before the project started. The ADRN model does not seem to work. Later, we were able to obtain what we need via UKDS. It seems that Secure Lab, own-site SRS, and Safe pods are the way forward.

Working with Next Steps (LSYPE) linked to NPD, it is clear that missing values in variables such as family income or parental education, even in cases that agreed to take part, bias the cases substantially. With each wave the dropout increase this bias in favour of higher-earning, more educated families. This makes the dataset far inferior to official datasets such as NPD (legally required, population data, longitudinal, and 600,000 new cases per year). Even using only the cases that are linked, predictions of edcucational outcomes using Next Steps and NPD are no better than those using the more limited indicators in NPD alone.

Findings from the 'trajectory' approach: The proportion of time spent in education while living below the poverty threshold (the years known to be FSM-eligible in England) would be a better indicator for allocating the pupil premium or contextualising school results than whether a student is currently or has ever been FSM-eligible. This would be fairer, and would help address the poverty gradient between schools, types of schools and regions. However, examining FSM status in sequence for every year that a child is in the school system is even better than both. Using the flag variable for FSM, a regression model for KS4 outcomes has an R of around 0.72, and using years FSM the same model has an R of around 0.82 (Gorard and Siddiqui 2018). Our new model is closer to 0.92. There are other changes to the models, but the biggest difference lies in using FSM and other indicators of disadvantage in this longitudinal way. The duration and precise pattern of childhood disadvantage matters when considering school outcomes, especially for FSM and SEN, more than the flag indicators suggest, and more than any other available indicator.

All attainment should be age-standardised even for entry to HE. Young children in multi-form entry schools can be taught in age-adjusted classes, as is current practice in some early-years settings, to minimise the summer-born problem. And the labelling of SEN (and to some extent first language) needs re-visiting in relation to age-in-year. Currently, immaturity is being mistaken as an educational problem. Where young people have serious and chronic special education needs, it needs to be realised more clearly that whatever support is being provided is currently not sufficient to reduce the attainment gap with their mainstream peers.

In our model, the coefficients for ethnic group (major) are small, and removing these flag variables makes very little difference to R in each year. The values for each ethnic minority are always small and almost always positive. In terms of attainment and progress at school, ethnic minority status as measured here, is not a determinant. There are raw-score ethnic attainment gaps, but they may be a proxy for other determinants such as SES and first language, plus temporary unfamiliarity with the system for new arrivals in England. Disaggregating Black Caribbean and Black African or other pupils might alter this picture. But no equivalent longitudinal analysis of a full age cohort has been done with theses more detailed categories. For the present, we must assume that average attainment and school progress are the same for pupils who differ only in terms of their ethnicity.

Having a first language other than English is similarly no long-term barrier to progress. In general, EAL pupils in Year 1 appear to obtain slightly higher KS4 scores than their peers. The problem, if there is one, may be short term, which is why pupils (still) labelled EAL in Year 11 appear to do slightly worse than their peers.

The importance of missing data: The coefficients for missing disadvantage data are larger than for ethnic group. In Year 1, the largest coefficient for ethnic group was 0.027 for 'Other', while missing FSM data had a coefficient of -0.035, and SEN missing was -0.073. Neither is large, and the coefficient for missing FSM data reduces with each year, partly as the amount of missing data declines. The coefficient for missing SEN data grows in the primary years and is always an issue. Missing data is partly a result of simple mobility. Pupils arriving from other home countries or from outside the UK will not have data for earlier years at school. The same applies to pupils transferring from private schools in England. There is no reason to assume that any of these groups are disproportionately disadvantaged. They may have to adjust to a new curriculum, but the model outcome is 11 years after these coefficients from Year 1, and differences in curriculum are unlikely to have such a long-term impact. Some groups such as Travellers (not disaggregated as an ethnic minority in the data) may move schools regularly, and for them missing data could represent the official statistics still catching up with the changes. As shown in Table 18 for Year 11 only, moving between schools is associated on average with a small penalty in terms of KS4 outcomes. Some groups may be refugees who do not have the necessary documentation in order to be known to be eligible for free school meals. Again, these would tend to be at least temporarily disadvantaged. Where young children have a learning challenge or disability that has been undiagnosed so far, meaning that assistance is not made available, this could reduce their chances of progress in the early years, and so of higher attainment at KS4.

Missing ethnicity or first language data in the early years is not linked to lower outcomes, presumably because neither factor is key and so unidentified EAL, for example, is not linked to lower progress once all other factors are taken into account. Nevertheless, missing data matters, and for FSM and SEN is a serious concern.







School (composition) effects: The school mix effect on attainment and participation outcomes may be small (of the magnitude of 5% improvement in results). But it is clear that schools and other education institutions could easily become more mixed over time via direct policy interventions, and that they could gain that 5% bonus for overall outcomes both cheaply and easily. There is a role for schools in reducing or at least not enlarging the poverty gradient. There is no benefit from clustering students in different schools and colleges in any way, whether by ability, faith or social class. There are only disadvantages for the system as a whole.
The model is not able to decide definitively whether there is a peer effect at school level. The model does clarify that the FSM segregation residual for each school is a better predictor of outcomes than the simple percentage of FSM-eligible pupils. Segregation here is the distance from a fair or evenly distributed school intake, assessed by the GS index. As with longitudinal measures of disadvantage, it would be better for future consideration of school intakes to use the segregation residual. However, the total R increase due to school (and area) factors in each year is small, almost negligible in comparison to individual measures.
Whether phantom or not, the school-level variables tell a consistent story across the duration of schooling. Pupils do worse in schools with clusters of disadvantage or clusters of prior attainment. Put another way, if this composition is real then schools should be as mixed as possible both socially and academically. This could lead to improved outcomes of between 0.05 and 0.15 of a standard deviation for almost no cost. No new schools need to be built, no new teachers employed, no new buildings or resources, just a more even spread of pupil intakes than currently by changing the school allocation process over a number of years.

Area of residence: Once pupil intake characteristics and prior attainment are accounted for, there is no evidence here that schools in different economic regions have different outcomes for equivalent pupils. This is important because a lot of current education policy in England is concerned with surface regional differences in attainment and either crediting schools for their results, as in the London Challenge, blaming schools for their failure, as in comments to the Select Committee about the North South divide, or simply where more work is being proposed, such as in supposedly underperforming coastal areas. As with so much in education, policy-makers appear to be reacting to raw-score differences linked to differences in regional populations rather than the performance of pupils, schools and teachers. Poorer areas of England need investment and infrastructure, not better schools in particular. Education is not a cheap solution to this.

There is no benefit for pupils from living in an area that has retained grammar schools, and their counterparts - the neighbouring schools that lose the highest scoring pupils. This is not a better system than in other areas, and does not lead to greater attainment for equivalent pupils. Again, this has implications for current government policy, which is being misled by raw-score results. Pupils attending schools in local authorities other than their area of residence do not gain systematically higher outcomes. The coefficient for the deprivation of area of residence (IDACI score) in Year 1 is -0.132, whereas the coefficient for school segregation is -0.052. Removing IDACI as a predictor in each year does not reduce the total R of the model. Instead the coefficient for school segregation changes to -0.119 without any obvious changes in other school or area variables.

School type: There is no such thing as a substantial school type effect. Schools are largely defined by who attends them. Once that is accounted for, there is no great difference between the outcomes of any of them (coefficients of 0.002 to 0.008). Only special schools (coefficient 0.073) might offer any advantage to the subset of pupils with greatest need. But even this could be a phantom composition effect, and is not stable enough to base policy on. For example, it is not an argument against inclusion of SEN pupils in mainstream settings for other reasons. Every new administration in England seems to want to create a new type of school for only some pupils or some parts of the country. Recently it has been Academies, Free schools, Studios and UTCs, faith-based and selective schools. None have better results than community, comprehensive schools, and some cost a great deal more.

Overall, the findings mean that when policy-makers, advocates of the success of the London Challenge, the inspection regime Ofsted, awards committees and others use the pupil premium gap as a measure of success they are probably and unwittingly being very unfair. There is a problem for all such pupil premium attainment gap calculations caused by missing data, and because they take no account of the proportion of local residents using private schools (both influencing the calculation by their absence). They are also unfair because they do not take account of the threshold nature of FSM-eligibility. They are ignoring the variation within that category stratified by prior educational challenges like SEN and EAL, and then again by the qualification outcomes used to calculate the gap. Almost as importantly, our prior analysis shows that different areas have different proportions of types of FSM pupils. Heavily disadvantaged areas are likely to have more of the always FSM-eligible pupils, and this makes any comparison with other areas based on the pupil premium gap intrinsically invalid. This is in no way an argument against the pupil premium policy itself, but it does suggest that the impact of the policy needs a rather more robust evaluation than simply measuring changes in the pupil premium attainment gap.
Exploitation Route There are implications for research funders (of longitudinal datasets, and those concerned with data access and security), researchers, policy-makers and educational practitioners (see Key Findings above). The findings for all four audiences have been widely published (a book out in September 2018, two Gold Open Access papers, two further papers, an international book chapter, six policy/practitioner articles, and five Working Papers). We have made three presentations to policy-makers, three to practitioners, and 13 to researchers including in Finland and Italy, plus an invited debate on grammar schools. More have been arranged. The findings and implications have been widely reported and discussed in the media especially concerning the age penalty, absences, north:south divide, faith schools, and grammar schools. Over 400 directly relevant media stories and appearances of our work have been logged in 2017-18. Written Evidence has been submitted to the House of Commons Education Committee - Special educational needs and disabilities - and the Scottish Parliament Education and Skills Committee - Attainment and achievement of school aged children experiencing poverty. We have been invited to give the 2018 Caroline Been Memorial Lecture at the House of Commons, and to a round table on overcoming poverty at the Royal Society of Edinburgh.

Thinks tanks, pressure groups and MPs have already used our findings to support issues particular to them. The work has been cited in Hansard (11/7/16). And we will encourage this to continue.


DfE has already added missing FSM outcomes, and EverFSM to NPD. We hope that they and Ofsted will encourage greater use of trajectory summaries in contextualising school performance. We also hope that practitioners will be more aware of age in years when diagnosing and labelling SEN. The focus must be on long-term poverty and SEN (vs school types, regions, ethnicity and EAL) as key determinants. The underlying problems are better addressed via infrastructure like transport than education directly. However, education can arrange a more even spread of school intakes, by having fewer types of schools and more scientific rules for allocating places in the public interest.
Sectors Communities and Social Services/Policy,Education,Transport,Other

URL https://www.dur.ac.uk/research/directory/view/?mode=project&id=942
 
Description Schools are using the finer measures of deprivation to help understand the attainment and progress of their students in context. Has been been discussed in House of Commons, Lords, Scottish Parliament, and was used by opposition parties and unions during 2017 election. Invited to give The Caroline Benn Memorial Lecture: House of Commons, London, November 2018 Basis of successful Festival of Social Science event November 2018 with policy-makers and practitioners. Videos of overview and full Question Time session available at https://www.dur.ac.uk/dece/news/. Narrative impact report came second in BERA Impact and Engagement Award 2019. Led to BERA Blog - https://www.bera.ac.uk/blog/lets-make-education-fairer, an invited paper in Research Intelligence - https://www.bera.ac.uk/news/2018-public-engagement-impact-award-winners When this project started: • The link between disadvantage and attainment in education was underestimated; • Schools in NE deemed underperforming, with advisers suggesting no investment in infrastructure until this was sorted out; • Government planned to increase grammar school and faithbased numbers; • DfE claimed every missed schoolday affected results • Countries worldwide did not routinely monitor social segregation in school systems. By end, these had all changed, with links to our project as being influential in raising awareness, changing minds, or leading to new policies/practices. The project created more successful ways of assessing disadvantage using official data, based on 'trajectory' of individual indicators, taking missing data seriously. These have changed use/interpretation for MPs (p.22, https://tinyurl.com/y742jnp3), Scottish Parliament (p348, https://tinyurl.com/yab2z77s), and the Royal Society Edinburgh. The Hartlepool Fabian Society organised series of meetings "What can one town do to improve its education" predicated on reanalysis of local attainment using length of poverty as a predictor (https://www.hartlepoollabour.org.uk/event/hartlepool-fabians-what-can-one-town-do/). Schools and authorities wrote to request reanalysis of their own figures that they could present to Ofsted. Relative 'failure' of schools in the NE is an illusion (https://edexec.co.uk/freeschoolsmealsmeasuremasks- north-south-divide-in-pupil-poverty/) - affecting how schools in the area address low attainment (https://www.newstatesman.com/feeds?page=119&qt-ns_mobile_editors_picks=1), and widening participation to HE (https://www.citymetric.com/politics/regional-inequality-has-huge-impact-teenagersuniversity- chances-so-how-can-we-fix-it-4259). Different types of schools increase segregation but do not improve attainment. Increasing selection to schools was relevant to 2017 election, made national headlines (http://educationmediacentre.org/newsreactions/conservative-manifesto-education-pledges-fundingfsm- grammar-schools-expert-analysis/). Policy-makers took notice (including Lucy Powell MP, Peter Kyle MP), sharing/discussing the research results online. More notice taken by opposition than government (https://www.libdemvoice.org/was-michael-gove-right-51800.html). Raised by Lord Storey in consultation on Schools that Work for Everyone (https://tinyurl.com/ya5qv8r8), and Lyn Brown MP discussing social mobility (https://tinyurl.com/y72osaa2). Our evidence forms substantial part of Full Fact reports 2016/2017 (https://tinyurl.com/yayflj4d), and Houses of Parliament POST Note, on methodologically robust studies of state selective schooling (https://tinyurl.com/ya66ur4x). Led to blueprint for an incoming Labour government (https://comprehensivefuture.org.uk/campaignitems/ decision-time-plan-phase-selection/). Helped by our user group, we used many avenues for the widest possible audience for the results, their robustness, and implications. These included giving evidence to Select Committees, a POST for the Houses of Parliament library, Labour Party Conference fringe events, oral and written evidence to Social Mobility Commission for their 2018 State of the Nation Report. We gave the Caroline Benn Memorial Lecture: House of Commons. We wrote non-academic pieces on the issues for New Scientist, Public Finance, Public Sector Focus, Schools Week, Full Fact, the Conversation, the Question, and Children and Young People Now. Over 80 broadcast interviews during the project to BBC TV Breakfast, Sky TV News, ITV News, BBC Radio 4 6:00 News, More or Less, and Any Answers, BBC Five Live, BBC Radio 1 Newsbeat, BBC Radio 2, most BBC local radio stations, independent radio. These concerned social mobility, school types such as Academies, faith-based, grammar, the summer birth problem, and impact of absences. We gave over 70 press interviews or wrote pieces for newspapers, including the New Statesman, Guardian, Sunday Times, i, Independent, Observer, Daily Mirror, Sun, Times Educational Supplement, Times Higher Education, the Herald, Northern Echo, Scottish Sun, with international outlets picking these up. These additionally covered inequalities, the N/S divide, Pupil Premium gap, mistakes in diagnosing SEN, and Ofsted grading. We held a successful ESRC Festival of Social Science event for the general public, and spoke at user conferences including Research Ed, National Local Authority, Schools of Tomorrow, Inside Government Pupil Premium Westminster Briefing, UCL Teacher Summer School, and Schools North East. We debated on grammar schools with Peter Hitchens at FitzWilliam College, Cambridge. We made all publications open access, and put them out on Twitter. One day eight MPs were tweeting about our work. We wrote blogs in English and Urdu. Our new approach led to different ways of assessing segregation/clustering of opportunities. We hosted scholars from Brazil, Spain and tribal regions of India wanting to learn about measuring socioeconomic segregation between schools/areas, using Gorard Segregation Index (GS). There is now monitoring of segregation using our ideas in Belgium, Spain, Portugal, Pakistan, South Africa, Texas, Columbia, Chile, Brazil. Work on attainment at school fed into debates about impact of absence from school (http://educationmediacentre.org/blog/does-missing-one-week-of-school-lead-to-lower-grades-byprofessor- stephen-gorard/), and summer born problem (https://www.parliament.uk/business/committees/committees-a-z/commons-select/educationcommittee/ dfe-evidence-check-forum/summer-born-children/?page=4), such as Full Fact 2016 report on lack of clear link between absences and attainment (https://tinyurl.com/y99skkef). Evidence raised in the House by Steve Double MP (https://tinyurl.com/y7pdusnl). We provided evidence for Parents' Union (http://www.theparentsunion.uk/wp-content/uploads/2017/11/statement-for-schools-nov-17.pdf). Fines for brief absences came under review (https://www.bbc.co.uk/news/education-38774418). Funded as part of ESRC Festival to discuss the findings and implications with the general public. Videos available of overview (https://youtu.be/BNNBSJlwcWs) and "question time" (https://youtu.be/5dkV8uNenbU). Q24 includes letters from schools seeking information on their own attainment gap, and explaining how the project (summarised in 2018 book "Education Policy") provoked powerful thinking about low attainment. There are examples from Spain, Brazil of international use of our approaches, and their relevance for overcoming local segregation of opportunities. We helped check the education manifesto "Reclaiming education", provided evidence for Chief Inspector for Schools, Campaign for State Education, Comprehensive Future. Letters explain that the influence of "Gorard and Siddiqui's research on public opinion concerning grammar schools policy cannot be overstated". Relating to NE purported underperformance, there are letters from Schools NE, and Pupil Premium Champion Northumberland, explaining how the work is used to reassure schools/parents, help decide where to address problems, and choose initiatives (e.g. via Opportunity North East), prepare briefing papers for ministers, and governor training.
First Year Of Impact 2017
Sector Education
Impact Types Cultural,Policy & public services