The impact of impact: How research value is created and assessed in the UK, US and Australia
Lead Research Organisation:
King's College London
Department Name: The Policy Institute
Abstract
Context and aims
As universities become bound up in a public role as drivers of economic prosperity, there has been a global policy shift towards measuring the value and benefit of publicly funded research. Governments have invested substantial resources in assessing research impact despite contestation around definitions, methodologies and processes. The rapid channelling of resources into impact has resulted in an under-theorised field that is largely dominated by pragmatic research policy concerns (i.e. focused on procedures and mechanisms). What is missing is a more nuanced conceptualisation of the societal value of research impact, which is essential in ensuring that research investment leads to meaningful societal outcomes. To capture this value, theoretically informed empirical insights into impact and the effects of its assessment on research cultures are needed. This, in turn, can inform the responsible evaluation of publicly funded research.
This will be the first large-scale comparative study examining research impact across countries. The project will compare academic cultures in three countries involved in the international push for the assessment of research impact; the UK, US and Australia, which utilise substantially different assessment models. The project will develop a theoretical understanding of research impact and its measurement. It will identify the effects of national research impact assessment policies and practices, and ascertain how they shape the ways research is done and their consequences for knowledge.
Potential applications and benefits
There is a pressing need for in-depth studies of the institutional and individual effects of research evaluation to interrogate frequently uncritical acceptance of dominant forms of valuation. This study will be novel in its examination of the powerful rhetoric of value that dominates discussions of research impact, and in its articulation of the challenges involved in the audit and commodification of academic research. This research will advance sociological and political understandings of research value, and will contribute theoretically informed knowledge to the applied fields of research evaluation and public policy. It will examine the emerging cultural patterns that surround research impact, and shed light on the political processes of decision-making and authority around impact policies. The study will shape the academic debate on the impact cultures of the UK, US and Australia, and contribute to a larger debate on the value of publicly funded research.
The study has strong potential to influence the policies that determine how impact is measured and how publicly funded research is governed in the UK, US, Australia and globally. By working directly with stakeholders including science policymakers, research funders and researchers, it will inform debate on the role of universities and research in society and facilitate more robust, reflexive and ethical research systems. It will provide recommendations on the structures and resources that policymakers, funders and higher education leaders could make available to researchers to support them in achieving high-quality research that makes a meaningful contribution to society. Understanding research impact and its assessment is of critical importance, not only strategically for UK, US and Australian governments, but also for countries with similar or emerging evaluation and funding models. It will allow countries to be intentional about creating responsive, equitable research systems that promote core values, such as a dual focus on research excellence and societal benefit.
Outputs from the study will include dissemination to policymakers, practitioners and the wider public through events for stakeholders, a working paper, a policy brief, blogs and media articles and a project website. Academic dissemination will occur through conference participation and open access articles in top journals.
As universities become bound up in a public role as drivers of economic prosperity, there has been a global policy shift towards measuring the value and benefit of publicly funded research. Governments have invested substantial resources in assessing research impact despite contestation around definitions, methodologies and processes. The rapid channelling of resources into impact has resulted in an under-theorised field that is largely dominated by pragmatic research policy concerns (i.e. focused on procedures and mechanisms). What is missing is a more nuanced conceptualisation of the societal value of research impact, which is essential in ensuring that research investment leads to meaningful societal outcomes. To capture this value, theoretically informed empirical insights into impact and the effects of its assessment on research cultures are needed. This, in turn, can inform the responsible evaluation of publicly funded research.
This will be the first large-scale comparative study examining research impact across countries. The project will compare academic cultures in three countries involved in the international push for the assessment of research impact; the UK, US and Australia, which utilise substantially different assessment models. The project will develop a theoretical understanding of research impact and its measurement. It will identify the effects of national research impact assessment policies and practices, and ascertain how they shape the ways research is done and their consequences for knowledge.
Potential applications and benefits
There is a pressing need for in-depth studies of the institutional and individual effects of research evaluation to interrogate frequently uncritical acceptance of dominant forms of valuation. This study will be novel in its examination of the powerful rhetoric of value that dominates discussions of research impact, and in its articulation of the challenges involved in the audit and commodification of academic research. This research will advance sociological and political understandings of research value, and will contribute theoretically informed knowledge to the applied fields of research evaluation and public policy. It will examine the emerging cultural patterns that surround research impact, and shed light on the political processes of decision-making and authority around impact policies. The study will shape the academic debate on the impact cultures of the UK, US and Australia, and contribute to a larger debate on the value of publicly funded research.
The study has strong potential to influence the policies that determine how impact is measured and how publicly funded research is governed in the UK, US, Australia and globally. By working directly with stakeholders including science policymakers, research funders and researchers, it will inform debate on the role of universities and research in society and facilitate more robust, reflexive and ethical research systems. It will provide recommendations on the structures and resources that policymakers, funders and higher education leaders could make available to researchers to support them in achieving high-quality research that makes a meaningful contribution to society. Understanding research impact and its assessment is of critical importance, not only strategically for UK, US and Australian governments, but also for countries with similar or emerging evaluation and funding models. It will allow countries to be intentional about creating responsive, equitable research systems that promote core values, such as a dual focus on research excellence and societal benefit.
Outputs from the study will include dissemination to policymakers, practitioners and the wider public through events for stakeholders, a working paper, a policy brief, blogs and media articles and a project website. Academic dissemination will occur through conference participation and open access articles in top journals.
Organisations
Publications
Berman G
(2025)
The benefits of being between (many) fields: Mapping the high-dimensional space of AI research
in Big Data & Society
Cohen E
(2024)
Researcher identities and values in the impact agenda: the case of artificial intelligence academics
in Higher Education
Haunschild R
(2025)
The influence of public policy and administration expertise on policy: an empirical study
in Evidence & Policy
Williams K
(2022)
What counts: Making sense of metrics of research value
in Science and Public Policy
Williams K
(2025)
Repertoires of research value: performing societal impact across countries
in Research Evaluation
Williams K
(2021)
Understanding, measuring, and encouraging public policy research impact
in Australian Journal of Public Administration
Williams K
(2023)
Investigating hybridity in artificial intelligence research
in Big Data & Society
Williams K
(2022)
Hybrid knowledge production and evaluation at the World Bank
in Policy and Society
Williams K
(2023)
Exploring the application of machine learning to expert evaluation of research impact.
in PloS one
| Description | Key findings: The cultural impact of impact. While impact has become central to how research is valued, its adoption is uneven across countries, institutions and disciplines. Many researchers see societal benefit as an important aspect of their work, but this does not necessarily translate into a uniform approach to assessing research excellence or success. The cultural impact of impact - that is, how impact policies shape research cultures - is mediated by the existing policy and incentive structures as well as by local disciplinary cultures and institutional logics. For example, although our interviews focused only on researchers with an artificial intelligence (AI) focus, we found a significant divide between those engaged in foundational theoretical work and those who develop and apply research in more practical settings. These groups differ in how they define impact, their perceptions of applied research, and how they integrate impact into their work. We also found that 'stronger' impact policies do not necessarily translate into comparatively stronger cultural orientation to impact amongst researchers, and in some cases can create more resistance to impact. Moreover, when impact becomes a marker of prestige, it can sharpen pre-existing tensions in a field, particularly around debates about the core values and purpose of a field. These findings highlight the need for flexible and context-sensitive research policies that align with the diverse ways academics engage with impact. New questions: 1. Our findings have raised new questions about the role of research impact policies in the context of globalisation. Impact policies may be seen as a response by national governments to assert control over the inherently global activity of science. This insight prompted a proposal of a special issue (expected 2025) of Research Evaluation around this, with invited contributing authors from several continents. 2. Our findings also raised the important question of how impact is shaping the educational role of academics. Our interview participants aim to integrate their impact and educational roles rather than seeing them as separate, prompting us to raise the possibility of a 'research-teaching-impact' nexus that is currently under-theorised. New networks: Kate Williams has held visiting fellowships at three international institutions: Humboldt University, Columbia University and New York University. Non-academic audiences have been reached through the production of policy reports and both formal and informal advice for the Chief Scientist of Australia, the Royal Society, Clarivate, Hong Kong's University Grants Committee, the UK Department for Science Innovation and Technology, four invited talks to policy audiences, and consulting on the new University Research Impact Framework implemented by the Chancellery and the Research Strategy for the Faculty of Arts at the University of Melbourne. |
| Exploitation Route | The outcomes are likely to be used by those involved in academic research policy and management at national and institutional levels. The findings are relevant to ongoing efforts to facilitate changes in academic culture that promote impact without creating negative unintended consequences. Further development and use of our outcomes could include increased focus on the challenges to early career researchers of impact policies since these individuals sometimes feel pressured to become expert in impact only to find it is unlikely to help lead to a secure post, and some researchers say it even holds careers back in some instances. We have also, arguably for the first time, empirically demonstrated impact as a potential source of significant tension within subfields, sharpening differences of opinion on the core purpose of a field. This will be relevant for those interested in the future of impact policy in the countries studied and more broadly. Our outcomes also point to the promise of more systematically seeking synergies between research, teaching and impact. Lastly, since our empirical focus was AI scientists, our outcomes are especially relevant for those interested in public AI research. |
| Sectors | Education Other |
| Description | The project's impact is evolving. Non-academic audiences have been reached through the production of policy reports for the Chief Scientist of Australia, the Royal Society, Clarivate, Hong Kong's University Grants Committee, four invited talks to policy audiences, and consulting on the new University Research Impact Framework implemented by the Chancellery and the Research Strategy for the Faculty of Arts at the University of Melbourne. |
| Description | Cited in University of Melbourne Central Research Strategy |
| Geographic Reach | Australia |
| Policy Influence Type | Citation in other policy documents |
| Impact | Academic work was the basis for the design of University of Melbourne's Chancellery strategy, which informs how the University invests in and manages research impact. |
| Description | Participation in evaluation of the University Grants Committee (Hong Kong) Research Assessment Exercise |
| Geographic Reach | Asia |
| Policy Influence Type | Membership of a guideline committee |
| Description | Participation in policy research project for the Australia Academy of Social Sciences, 'Modernising research assessment in Australia' |
| Geographic Reach | Australia |
| Policy Influence Type | Contribution to a national consultation/review |
| Description | Participation in policy research project for the Royal Society, 'Taxonomy of Artificial Intelligence related technologies for scientific research' |
| Geographic Reach | National |
| Policy Influence Type | Contribution to a national consultation/review |
| Description | National research impact policies: Uncovering the 'value' in evaluation' |
| Amount | $164,764 (AUD) |
| Organisation | Australian Research Council |
| Sector | Public |
| Country | Australia |
| Start | 01/2024 |
| End | 12/2027 |
| Description | Wider societal value of research and consequences of its assessment: A multi-country and multi-method study (MultiSocVal) |
| Amount | € 1,597,400 (EUR) |
| Organisation | Volkswagen Foundation |
| Sector | Charity/Non Profit |
| Country | Germany |
| Start | 08/2024 |
| End | 08/2028 |
| Description | Expert workshop on the theory and practice of research impact in international perspective |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Other audiences |
| Results and Impact | An invited group of 12 guests, involving both other scholars of research impact as well as impact/evaluation professional practitioners, attended a two-day workshop to discuss the project's interim findings and share related ongoing work/research. This resulted in the attendees successfully submitting a special issue proposal at the journal Research Evaluation (expected publication 2025), which has a reputation of reaching both research and practitioner audiences. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Invited to UKRI-DSIT roundtable to discuss effective ways of supporting and funding R&D. |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Supporters |
| Results and Impact | Invited to roundtable to contribute to discussion on how to best support and fund R&D, partly based on knowledge/expertise gained during this project. |
| Year(s) Of Engagement Activity | 2024 |
| Description | Presentation at Impactful Career Pathways Short Course, University of Melbourne |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Delivered talk to Impactful Career Pathways, drawing on Bourdieu's social theory as a framework to explore different forms of capital underpinning the five different facets of impact from scholarly and creative, to policy/government/law, to public debates, to practices and processes, and enterprise and commercialisation. |
| Year(s) Of Engagement Activity | 2023 |
| Description | Revisiting the role of research in society' public talk |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Public/other audiences |
| Results and Impact | Convened a public panel discussion on research impact called 'Revisiting the role of research in society' at the University of Melbourne, on 9 September 2021. The live virtual event drew over 80 attendees, with many more watching the recording. |
| Year(s) Of Engagement Activity | 2022 |
| Description | Talk at Westminster Higher Education Forum |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Policymakers/politicians |
| Results and Impact | Talk at Westminster Higher Education Forum on research impact. |
| Year(s) Of Engagement Activity | 2022 |
| Description | Talk at the Innovation to Research Impact Assessment, Virtual International Convention for Research Administrators |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | National |
| Primary Audience | Professional Practitioners |
| Results and Impact | Talk given at the Innovation to Research Impact Assessment, Virtual International Convention for Research Administrators, 1-3 March 2022, a virtual international event run across four continents. |
| Year(s) Of Engagement Activity | 2022 |
