The Potentials, Politics and Practices of International Educational Assessment
Lead Research Organisation:
University of East Anglia
Department Name: International Development
Abstract
In the last two decades, educational policy and practice have been heavily influenced by the increasing implementation of international educational assessments. The most influential contemporary international assessments include PISA (Programme for International Student Assessment carried; TIMSS (Trends in International Mathematics and Science Survey) and PIAAC (Programme for International Assessment of Adult Competencies). The OECD (Organisation for Economic Co-operation and Development) administers PISA and PIAAC, and has become an influential trans-national actor on global educational policy - within, and increasingly beyond OECD member countries.
The significance of international educational big data, the comparative methodology, its interpretation and impacts on policy and practice is increasingly viewed by scholars as a technology of trans-national governance - offering not only potentials for comparison and learning about successful educational systems, but also political questions about how such data frames and influences national and international educational policies (and how countries use and respond to such data), and challenging technical and conceptual issues associated with the practice of cross-cultural assessment and comparison (i.e. issues of test standardization and modification for use across culturally and linguistically diverse contexts).
International assessments have attracted considerable academic attention in recent years including a number of edited collections (e.g. Meyer and Benavot 2013; Pereyra, Kotthoff, et al. 2011; Engel and Williams 2013; and the 2013 and 2014 World Yearbooks in Education). However, the various aspects of the field (we call Potentials, Politics and Practices) have not yet been integrated into a shared field and framework. In this seminar series we intend to bring together the different elements and foci of international assessment into a combined field of 'International Assessment Studies' (Addey 2013).
The series will be run by a new network of researchers called the Laboratory of International Assessment. The network is UK based, but increasingly involves academic researchers on a global level to reflect the global interest in the phenomenon. There are of course academic networks and professional associations with an interest in these themes. What makes our network distinctive is our commitment to work with and involve international testing agencies, and to use 'insider accounts' (first had accounts of international assessment in practice) as the basis for investigation.
The aim of the seminar series is to provide rigorous meta-analysis on key policy themes, to inform academics, testing agencies and the end users of international assessment data. Our objectives are: to act as a vehicle for academic enquiry into the potentials, politics and practices and impacts of international educational assessments ; to develop a thematic agenda for the international assessment studies as an important new field of academic research and publication ; and to build a sustainable and robust international network of academics and practitioners around a set of common interests and concerns.
We will run a series of six (two day) seminars with a distinctly international orientation (including seminars in the UK, United States, Peru, and Brussels). This reflects the global nature of international assessments and their impacts. These seminars will include both public plenary lectures (that will be video recorded and made available on-line), and invited smaller events for invited participants. The seminars will help to build our research network, and because of the involvement of testing agencies and end-users of assessment data (including policy makers, educationalists, civil society organisations and media), we will have the potential for significant impact.
The significance of international educational big data, the comparative methodology, its interpretation and impacts on policy and practice is increasingly viewed by scholars as a technology of trans-national governance - offering not only potentials for comparison and learning about successful educational systems, but also political questions about how such data frames and influences national and international educational policies (and how countries use and respond to such data), and challenging technical and conceptual issues associated with the practice of cross-cultural assessment and comparison (i.e. issues of test standardization and modification for use across culturally and linguistically diverse contexts).
International assessments have attracted considerable academic attention in recent years including a number of edited collections (e.g. Meyer and Benavot 2013; Pereyra, Kotthoff, et al. 2011; Engel and Williams 2013; and the 2013 and 2014 World Yearbooks in Education). However, the various aspects of the field (we call Potentials, Politics and Practices) have not yet been integrated into a shared field and framework. In this seminar series we intend to bring together the different elements and foci of international assessment into a combined field of 'International Assessment Studies' (Addey 2013).
The series will be run by a new network of researchers called the Laboratory of International Assessment. The network is UK based, but increasingly involves academic researchers on a global level to reflect the global interest in the phenomenon. There are of course academic networks and professional associations with an interest in these themes. What makes our network distinctive is our commitment to work with and involve international testing agencies, and to use 'insider accounts' (first had accounts of international assessment in practice) as the basis for investigation.
The aim of the seminar series is to provide rigorous meta-analysis on key policy themes, to inform academics, testing agencies and the end users of international assessment data. Our objectives are: to act as a vehicle for academic enquiry into the potentials, politics and practices and impacts of international educational assessments ; to develop a thematic agenda for the international assessment studies as an important new field of academic research and publication ; and to build a sustainable and robust international network of academics and practitioners around a set of common interests and concerns.
We will run a series of six (two day) seminars with a distinctly international orientation (including seminars in the UK, United States, Peru, and Brussels). This reflects the global nature of international assessments and their impacts. These seminars will include both public plenary lectures (that will be video recorded and made available on-line), and invited smaller events for invited participants. The seminars will help to build our research network, and because of the involvement of testing agencies and end-users of assessment data (including policy makers, educationalists, civil society organisations and media), we will have the potential for significant impact.
Planned Impact
Who will benefit from the seminar series?
The non-academic beneficiaries in the seminar series will the institutions and individuals who are involved directly in international educational assessments, and the end users of assessment data.
The institutions and individuals who are directly involved in international assessments include the test administering agencies (e.g. the Organisation for Economic Co-operation and Development, OECD, and the International Association for the Evaluation of Educational Achievement, IEA), and government departments of education and statistics who administer assessment programmes. These institutions not only administer international assessments, but also present and respond to their findings.
A second group of non-academic beneficiaries include specialist testing agencies that are responsible for test and test item development and technical processes of test translation and adaptation, and the psychometric production of test data. These include the technical consortium partners of international testing programmes (e.g. such as Educational Testing Service, ETS; the Data Processing Centre at IEA, and CAPSTAN Linguistic Quality Control who manage test item translation and cross-cultural adaptation.
A final group of non-academic beneficiaries are the end users of international assessment data. This includes government departments of education, teacher training institutions, civil society organisations, government departments responsible for labour market competitiveness such as the UK government department for Business, Innovation and Skills, and the European Commission. The end users also include media organisations and journalists who report on the findings of assessment programmes, and the European Commission
How will they benefit?
The non-academic beneficiaries will benefit from deeper and more holistic understanding of the potentials, politics and practices of international assessment programmes. The holistic orientation of the seminar series includes multiple dimensions of the assessment process - from the design and adaptation of test materials, to systems of governance and accountability, the analysis and representation of assessment data, and the policy impacts.
The seminars will involve rigorous analysis of assessment practice, the development of meta-themes and concepts and the collaborative development of new agendas for research and innovation. This will include opportunities for new agendas and opportunities for research collaboration (i.e. including academics, testing agencies and the end users of assessment data).
The non-academic beneficiaries in the seminar series will the institutions and individuals who are involved directly in international educational assessments, and the end users of assessment data.
The institutions and individuals who are directly involved in international assessments include the test administering agencies (e.g. the Organisation for Economic Co-operation and Development, OECD, and the International Association for the Evaluation of Educational Achievement, IEA), and government departments of education and statistics who administer assessment programmes. These institutions not only administer international assessments, but also present and respond to their findings.
A second group of non-academic beneficiaries include specialist testing agencies that are responsible for test and test item development and technical processes of test translation and adaptation, and the psychometric production of test data. These include the technical consortium partners of international testing programmes (e.g. such as Educational Testing Service, ETS; the Data Processing Centre at IEA, and CAPSTAN Linguistic Quality Control who manage test item translation and cross-cultural adaptation.
A final group of non-academic beneficiaries are the end users of international assessment data. This includes government departments of education, teacher training institutions, civil society organisations, government departments responsible for labour market competitiveness such as the UK government department for Business, Innovation and Skills, and the European Commission. The end users also include media organisations and journalists who report on the findings of assessment programmes, and the European Commission
How will they benefit?
The non-academic beneficiaries will benefit from deeper and more holistic understanding of the potentials, politics and practices of international assessment programmes. The holistic orientation of the seminar series includes multiple dimensions of the assessment process - from the design and adaptation of test materials, to systems of governance and accountability, the analysis and representation of assessment data, and the policy impacts.
The seminars will involve rigorous analysis of assessment practice, the development of meta-themes and concepts and the collaborative development of new agendas for research and innovation. This will include opportunities for new agendas and opportunities for research collaboration (i.e. including academics, testing agencies and the end users of assessment data).
Publications
Addey C
(2021)
Passports to the Global South, UN flags, favourite experts: understanding the interplay between UNESCO and the OECD within the SDG4 context
in Globalisation, Societies and Education
Addey C
(2020)
Translating PISA, translating the world
in Comparative Education
Addey C
(2020)
Assembled validity: rethinking Kane's argument-based approach in the context of International Large-Scale Assessments (ILSAs)
in Assessment in Education: Principles, Policy & Practice
Addey C
(2017)
Golden relics & historical standards: how the OECD is expanding global education governance through PISA for Development
in Critical Studies in Education
Gorur R
(2017)
Towards productive critique of large-scale comparisons in education
in Critical Studies in Education
Gorur R
(2017)
Statistics and statecraft: exploring the potentials, politics and practices of international educational assessment
in Critical Studies in Education
Grek S
(2017)
Socialisation, learning and the OECD's Reviews of National Policies for Education : the case of Sweden
in Critical Studies in Education
Guadalupe C
(2017)
Standardisation and diversity in international assessments: barking up the wrong tree?
in Critical Studies in Education
Hamilton M
(2017)
How International Large-Scale Skills Assessments engage with national actors: mobilising networks through policy, media and public knowledge
in Critical Studies in Education
Lingard B
(2017)
Placing PISA and PISA for schools in two federalisms, Australia and the USA
in Critical Studies in Education
| Description | 1. Viability of the Research Network: It is a moment of growth for policy and research interest in International Large-Scale Assessments (ILSAs). The ESRC series generated unexpectedly high levels of interest internationally from academics and testing agencies including international and UK government actors and intergovernmental agencies, such as the OECD, UNESCO and UNESCO Institute of Statistics. We had participation from international testing agencies - as participants and as speakers in the seminars, and on our expert advisory group. The laboratory has continued to thrive, most recently (2020/21) with an internatinal webinar seires on '20 Years of ILSAs' - which has been attended by a large group of 100 participants including acaemics and practitioners. The details are on the Laboratory website: http://www.international-assessments.org 2. Identifying research interests between different groups: We had deliberately promoted engagement and collaboration between academic researchers, policy makers and testing agencies. This was novel, and welcomed by testing agencies and academics (video testimony, seminar 6). This engagement is not always supported by academic researchers who sometimes adopt an oppositional stance toward ILSAs, and alienate testing agencies. An oppositional stance has intellectual value -but does not make the most from academic-testing agency collaboration. Our stated commitment toward active engagement was clearly welcome, and generated sustained research links. It was clear that academics and testing agencies have shared interests at the forefront of research agendas. The OECD funded research as a direct result of participation in one of seminars (seminar 4). Maddox has since been invited to participate in high level OECD working groups (i.e. most recently the PISA innovation group). 3. Participation from policy makers was less easily achieved: This highlights an important area of learning (i.e. as a research finding). We did have good participation from international testing agencies (e.g. OECD). However, public and policy engagement with ILSAs and their data is an area that merits further research. Firstly, ILSAs are highly technical - and this means that policy makers and public institutions find it hard to understand and engage with the test results and methodology. ILSAs use public funds and often impact on national policies but without effective public, media and policy understanding about how the data are generated or the policy implications. This was identified at seminar 5 at Lancaster - on the topic of the media and public reception. This suggests that further research on policy and public reception of ILSA methods and data would be valuable. Secondly, there is a complex relationship between national and international educational assessments, their data and accountability. It is not always clear how those studies speak to each other, or how effective governance and accountability of ILSAs should take place (e.g. themes from seminar 1). We pursued this theme at our Laboratory seminar held at and hosted by Deakin University, Melbourne in December 2017. 4. Emergent themes: In addition to the themes identified (e.g. public engagement, national and international engagement, accountability and governance), the series identified other research themes that we will follow up in future research seminars. These include a theme about the impacts of ILSAs on low income countries (Seminar 3); the involvement of private sector actors in ILAs (seminar 6); and on the ethics and consequences of ILSAs (from seminar 4). We will pursue each of these themes in future seminars and research applications. We also discovered a new theme related to psychological investigations into ILSA performance that (from seminar 4) that have received funding from OECD and continue to generate interest from testing organizations. 5. As a result of the seminar series we identified potential commercial value (as a spin-out company) and "Assessment MicroAnalyticsLtd" was launched on 1st October 2018 (www.microanalytics.co.uk). We began to discover that potential as a result of the information and networks that were established during the seminar series (especially the seminar on data collection held at UEA). The company has grown since its launch, and now works on large-scale educational assessments with national ministries of education (France, Luxembourg), and commercial testing organisations and exam boards in the UK and the United States (Pearson, AQA, ACT). |
| Exploitation Route | We will continue to promote the intellectual agenda through the research network. This will include seminars, research and publications. The Series generated a strong international research network managed by the Laboratory of International Assessment Studies (www.international-assessments.org). Two follow up seminars (Deakin on National and International Assessments, held December 2017, and Manchester (on Ethics and Assessment) planned for September 2019), and our recent (20/21) webinar series on '20 Years of ILSAs'. The details of our seminars are publicized on our Lab website. Blogs and video recordings of ESRC seminar presentations are on YouTube and our Laboratory website. Some of these video presentations are being accessed and cited in publications (per com. Zumbo). We will pursue the research themes through research, and collaboration on research applications including applied collaboration with international testing agencies. We already have on-going spin-off research that can be attributed to the success of the seminar series. This includes new research by Gorur (Co-I on the series) on Global Policy Networks and Accountability in the Indo-Pacific; Addey (Co-I on the series) on PISA for Development; Maddox (and Bayliss, UEA) received OECD funded applied research on ILSAs that is a direct result of the seminars. We have disseminated our academic findings through publications - Maddox (Ed) (2018) 'International Large-Scale Assessments in Education' (Bloomsbury), and Gorur (Ed) (2017) special issue of 'Critical Studies in Education' with contributions from participants in the ESRC Seminar Series. |
| Sectors | Digital/Communication/Information Technologies (including Software),Education |
| URL | http://microanalytics.co.uk |
| Description | The seminars, symposiums and webinars ran by the Laboratory of International Assessment Studies, and our associate publications have contributed to policy making processes at UNESCO and the OECD, where Co-I's from the seminar series have been involved in policy and practice oriented events and working groups. Addey (co-I) has been commissioned to write papers for UNESCO. The impact of Maddox's research on international large-scale assessments at UNESCO is noted in their policy publications in 2019 and 2020. Maddox has been commissioned to write a paper and to contribute to assessment innovation processes at the OECD. Maddox's research contribued directly to the OECD programmes in large-scale assessment (PIAAC and PISA for Development). The spinout company, Assessment MicroAnalytics Ltd has impacted directly on the design and administration of international and national large-scale asessments with impacts in at least 40 countries. Maddox has also been invited to join, and contributed to the research committee for AQA (UK exam board) since 2020. Maddox remains involved in the Assessment Innovation process with the OECD. These impacts are described in a forthcoming REF impact case study. |
| First Year Of Impact | 2018 |
| Sector | Digital/Communication/Information Technologies (including Software),Education |
| Impact Types | Economic,Policy & public services |
| Description | Response processes, interaction and test taker engagement OECD Large-Scale Assessments |
| Organisation | Organisation for Economic Co-operation and Development OECD |
| Country | France |
| Sector | Public |
| PI Contribution | This partnership started at the ESRC seminar on 'The production of data in international assessments' held at UEA Norwich in 2015. At that event, as a direct impact of the seminar presentations and discussions, the OECD representative agreed to fund exploratory Lab based research at UEA on 'The relationship between physiological response, interaction and test taker engagement in the OECD PIAAC Assessment. The OECD provided £16,447 and Bryan Maddox (UEA) and Andrew Bayliss (UEA) were PI's on that research. The research was completed in 2017, and our first publication from the research 'Observing Response Processes with Eye Tracking in International Large-Scale Assessments: Evidence from the OECD PIAAC assessment ' (authors, Maddox, B. Bayliss, A; Flemming, P; Engelhardt, P, & Bourgonovi, F). has just been accepted to the European Journal of Psychology of Education for a Special Issue on 'Large scale assessments and insights into teaching and learning practices'. Bourgonovi (co-author) is a member of OECD staff. A team of 3 staff from the OECD visited UEA in July 2017 to participate in a dissemination workshop and discussion of the research findings. Maddox also visited OECD Paris in September 2017 (Using University managed ESRC Impact Accelerator funds) to discuss the research and potential follow up research. Follow up research was commissioned in 2019 on the OECD PISA for Development Programme (fieldwork in Senegal), and on the OECD PIAAC Assessment in the UK in 2020. Maddox was invited to contribute to the OECD innovation process for PISA in 2020 and 2021. |
| Collaborator Contribution | UEA provided Laboratory space and staff and managed the research. OECD provided their test materials (a 'virtual machine') and technical support. |
| Impact | Maddox, B. Bayliss, A; Flemming, P; Engelhardt, P, & Bourgonovi, F (under review). 'Observing Response Processes with Eye Tracking in International Large-Scale Assessments: Evidence from the OECD PIAAC assessment '. European Journal of Psychology of Education. Special Issue on 'Large scale assessments and insights into teaching and learning practices'. This is a multi-disciplinary collaboration including anthropology, psychology and education. |
| Start Year | 2015 |