📣 Help Shape the Future of UKRI's Gateway to Research (GtR)

We're improving UKRI's Gateway to Research and are seeking your input! If you would be interested in being interviewed about the improvements we're making and to have your say about how we can make GtR more user-friendly, impactful, and effective for the Research and Innovation community, please email gateway@ukri.org.

The impact of impact: How research value is created and assessed in the UK, US and Australia

Lead Research Organisation: King's College London
Department Name: The Policy Institute

Abstract

Context and aims

As universities become bound up in a public role as drivers of economic prosperity, there has been a global policy shift towards measuring the value and benefit of publicly funded research. Governments have invested substantial resources in assessing research impact despite contestation around definitions, methodologies and processes. The rapid channelling of resources into impact has resulted in an under-theorised field that is largely dominated by pragmatic research policy concerns (i.e. focused on procedures and mechanisms). What is missing is a more nuanced conceptualisation of the societal value of research impact, which is essential in ensuring that research investment leads to meaningful societal outcomes. To capture this value, theoretically informed empirical insights into impact and the effects of its assessment on research cultures are needed. This, in turn, can inform the responsible evaluation of publicly funded research.

This will be the first large-scale comparative study examining research impact across countries. The project will compare academic cultures in three countries involved in the international push for the assessment of research impact; the UK, US and Australia, which utilise substantially different assessment models. The project will develop a theoretical understanding of research impact and its measurement. It will identify the effects of national research impact assessment policies and practices, and ascertain how they shape the ways research is done and their consequences for knowledge.

Potential applications and benefits

There is a pressing need for in-depth studies of the institutional and individual effects of research evaluation to interrogate frequently uncritical acceptance of dominant forms of valuation. This study will be novel in its examination of the powerful rhetoric of value that dominates discussions of research impact, and in its articulation of the challenges involved in the audit and commodification of academic research. This research will advance sociological and political understandings of research value, and will contribute theoretically informed knowledge to the applied fields of research evaluation and public policy. It will examine the emerging cultural patterns that surround research impact, and shed light on the political processes of decision-making and authority around impact policies. The study will shape the academic debate on the impact cultures of the UK, US and Australia, and contribute to a larger debate on the value of publicly funded research.

The study has strong potential to influence the policies that determine how impact is measured and how publicly funded research is governed in the UK, US, Australia and globally. By working directly with stakeholders including science policymakers, research funders and researchers, it will inform debate on the role of universities and research in society and facilitate more robust, reflexive and ethical research systems. It will provide recommendations on the structures and resources that policymakers, funders and higher education leaders could make available to researchers to support them in achieving high-quality research that makes a meaningful contribution to society. Understanding research impact and its assessment is of critical importance, not only strategically for UK, US and Australian governments, but also for countries with similar or emerging evaluation and funding models. It will allow countries to be intentional about creating responsive, equitable research systems that promote core values, such as a dual focus on research excellence and societal benefit.

Outputs from the study will include dissemination to policymakers, practitioners and the wider public through events for stakeholders, a working paper, a policy brief, blogs and media articles and a project website. Academic dissemination will occur through conference participation and open access articles in top journals.
 
Description Key findings: The cultural impact of impact.

While impact has become central to how research is valued, its adoption is uneven across countries, institutions and disciplines. Many researchers see societal benefit as an important aspect of their work, but this does not necessarily translate into a uniform approach to assessing research excellence or success. The cultural impact of impact - that is, how impact policies shape research cultures - is mediated by the existing policy and incentive structures as well as by local disciplinary cultures and institutional logics. For example, although our interviews focused only on researchers with an artificial intelligence (AI) focus, we found a significant divide between those engaged in foundational theoretical work and those who develop and apply research in more practical settings. These groups differ in how they define impact, their perceptions of applied research, and how they integrate impact into their work. We also found that 'stronger' impact policies do not necessarily translate into comparatively stronger cultural orientation to impact amongst researchers, and in some cases can create more resistance to impact. Moreover, when impact becomes a marker of prestige, it can sharpen pre-existing tensions in a field, particularly around debates about the core values and purpose of a field.
These findings highlight the need for flexible and context-sensitive research policies that align with the diverse ways academics engage with impact.

New questions:
1. Our findings have raised new questions about the role of research impact policies in the context of globalisation. Impact policies may be seen as a response by national governments to assert control over the inherently global activity of science. This insight prompted a proposal of a special issue (expected 2025) of Research Evaluation around this, with invited contributing authors from several continents.

2. Our findings also raised the important question of how impact is shaping the educational role of academics. Our interview participants aim to integrate their impact and educational roles rather than seeing them as separate, prompting us to raise the possibility of a 'research-teaching-impact' nexus that is currently under-theorised.

New networks:
Kate Williams has held visiting fellowships at three international institutions: Humboldt University, Columbia University and New York University. Non-academic audiences have been reached through the production of policy reports and both formal and informal advice for the Chief Scientist of Australia, the Royal Society, Clarivate, Hong Kong's University Grants Committee, the UK Department for Science Innovation and Technology, four invited talks to policy audiences, and consulting on the new University Research Impact Framework implemented by the Chancellery and the Research Strategy for the Faculty of Arts at the University of Melbourne.
Exploitation Route The outcomes are likely to be used by those involved in academic research policy and management at national and institutional levels. The findings are relevant to ongoing efforts to facilitate changes in academic culture that promote impact without creating negative unintended consequences. Further development and use of our outcomes could include increased focus on the challenges to early career researchers of impact policies since these individuals sometimes feel pressured to become expert in impact only to find it is unlikely to help lead to a secure post, and some researchers say it even holds careers back in some instances. We have also, arguably for the first time, empirically demonstrated impact as a potential source of significant tension within subfields, sharpening differences of opinion on the core purpose of a field. This will be relevant for those interested in the future of impact policy in the countries studied and more broadly. Our outcomes also point to the promise of more systematically seeking synergies between research, teaching and impact.
Lastly, since our empirical focus was AI scientists, our outcomes are especially relevant for those interested in public AI research.
Sectors Education

Other

 
Description The project's impact is evolving. Non-academic audiences have been reached through the production of policy reports for the Chief Scientist of Australia, the Royal Society, Clarivate, Hong Kong's University Grants Committee, four invited talks to policy audiences, and consulting on the new University Research Impact Framework implemented by the Chancellery and the Research Strategy for the Faculty of Arts at the University of Melbourne.
 
Description Cited in University of Melbourne Central Research Strategy
Geographic Reach Australia 
Policy Influence Type Citation in other policy documents
Impact Academic work was the basis for the design of University of Melbourne's Chancellery strategy, which informs how the University invests in and manages research impact.
 
Description Participation in evaluation of the University Grants Committee (Hong Kong) Research Assessment Exercise
Geographic Reach Asia 
Policy Influence Type Membership of a guideline committee
 
Description Participation in policy research project for the Australia Academy of Social Sciences, 'Modernising research assessment in Australia'
Geographic Reach Australia 
Policy Influence Type Contribution to a national consultation/review
 
Description Participation in policy research project for the Royal Society, 'Taxonomy of Artificial Intelligence related technologies for scientific research'
Geographic Reach National 
Policy Influence Type Contribution to a national consultation/review
 
Description National research impact policies: Uncovering the 'value' in evaluation'
Amount $164,764 (AUD)
Organisation Australian Research Council 
Sector Public
Country Australia
Start 01/2024 
End 12/2027
 
Description Wider societal value of research and consequences of its assessment: A multi-country and multi-method study (MultiSocVal)
Amount € 1,597,400 (EUR)
Organisation Volkswagen Foundation 
Sector Charity/Non Profit
Country Germany
Start 08/2024 
End 08/2028
 
Description Expert workshop on the theory and practice of research impact in international perspective 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact An invited group of 12 guests, involving both other scholars of research impact as well as impact/evaluation professional practitioners, attended a two-day workshop to discuss the project's interim findings and share related ongoing work/research. This resulted in the attendees successfully submitting a special issue proposal at the journal Research Evaluation (expected publication 2025), which has a reputation of reaching both research and practitioner audiences.
Year(s) Of Engagement Activity 2023
 
Description Invited to UKRI-DSIT roundtable to discuss effective ways of supporting and funding R&D. 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Supporters
Results and Impact Invited to roundtable to contribute to discussion on how to best support and fund R&D, partly based on knowledge/expertise gained during this project.
Year(s) Of Engagement Activity 2024
 
Description Presentation at Impactful Career Pathways Short Course, University of Melbourne 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Delivered talk to Impactful Career Pathways, drawing on Bourdieu's social theory as a framework to explore different forms of capital underpinning the five different facets of impact from scholarly and creative, to policy/government/law, to public debates, to practices and processes, and enterprise and commercialisation.
Year(s) Of Engagement Activity 2023
 
Description Revisiting the role of research in society' public talk 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Convened a public panel discussion on research impact called 'Revisiting the role of research in society' at the University of Melbourne, on 9 September 2021. The live virtual event drew over 80 attendees, with many more watching the recording.
Year(s) Of Engagement Activity 2022
 
Description Talk at Westminster Higher Education Forum 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Talk at Westminster Higher Education Forum on research impact.
Year(s) Of Engagement Activity 2022
 
Description Talk at the Innovation to Research Impact Assessment, Virtual International Convention for Research Administrators 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Talk given at the Innovation to Research Impact Assessment, Virtual International Convention for Research Administrators, 1-3 March 2022, a virtual international event run across four continents.
Year(s) Of Engagement Activity 2022