The impact of impact: How research value is created and assessed in the UK, US and Australia
Lead Research Organisation:
King's College London
Department Name: The Policy Institute
Abstract
Context and aims
As universities become bound up in a public role as drivers of economic prosperity, there has been a global policy shift towards measuring the value and benefit of publicly funded research. Governments have invested substantial resources in assessing research impact despite contestation around definitions, methodologies and processes. The rapid channelling of resources into impact has resulted in an under-theorised field that is largely dominated by pragmatic research policy concerns (i.e. focused on procedures and mechanisms). What is missing is a more nuanced conceptualisation of the societal value of research impact, which is essential in ensuring that research investment leads to meaningful societal outcomes. To capture this value, theoretically informed empirical insights into impact and the effects of its assessment on research cultures are needed. This, in turn, can inform the responsible evaluation of publicly funded research.
This will be the first large-scale comparative study examining research impact across countries. The project will compare academic cultures in three countries involved in the international push for the assessment of research impact; the UK, US and Australia, which utilise substantially different assessment models. The project will develop a theoretical understanding of research impact and its measurement. It will identify the effects of national research impact assessment policies and practices, and ascertain how they shape the ways research is done and their consequences for knowledge.
Potential applications and benefits
There is a pressing need for in-depth studies of the institutional and individual effects of research evaluation to interrogate frequently uncritical acceptance of dominant forms of valuation. This study will be novel in its examination of the powerful rhetoric of value that dominates discussions of research impact, and in its articulation of the challenges involved in the audit and commodification of academic research. This research will advance sociological and political understandings of research value, and will contribute theoretically informed knowledge to the applied fields of research evaluation and public policy. It will examine the emerging cultural patterns that surround research impact, and shed light on the political processes of decision-making and authority around impact policies. The study will shape the academic debate on the impact cultures of the UK, US and Australia, and contribute to a larger debate on the value of publicly funded research.
The study has strong potential to influence the policies that determine how impact is measured and how publicly funded research is governed in the UK, US, Australia and globally. By working directly with stakeholders including science policymakers, research funders and researchers, it will inform debate on the role of universities and research in society and facilitate more robust, reflexive and ethical research systems. It will provide recommendations on the structures and resources that policymakers, funders and higher education leaders could make available to researchers to support them in achieving high-quality research that makes a meaningful contribution to society. Understanding research impact and its assessment is of critical importance, not only strategically for UK, US and Australian governments, but also for countries with similar or emerging evaluation and funding models. It will allow countries to be intentional about creating responsive, equitable research systems that promote core values, such as a dual focus on research excellence and societal benefit.
Outputs from the study will include dissemination to policymakers, practitioners and the wider public through events for stakeholders, a working paper, a policy brief, blogs and media articles and a project website. Academic dissemination will occur through conference participation and open access articles in top journals.
As universities become bound up in a public role as drivers of economic prosperity, there has been a global policy shift towards measuring the value and benefit of publicly funded research. Governments have invested substantial resources in assessing research impact despite contestation around definitions, methodologies and processes. The rapid channelling of resources into impact has resulted in an under-theorised field that is largely dominated by pragmatic research policy concerns (i.e. focused on procedures and mechanisms). What is missing is a more nuanced conceptualisation of the societal value of research impact, which is essential in ensuring that research investment leads to meaningful societal outcomes. To capture this value, theoretically informed empirical insights into impact and the effects of its assessment on research cultures are needed. This, in turn, can inform the responsible evaluation of publicly funded research.
This will be the first large-scale comparative study examining research impact across countries. The project will compare academic cultures in three countries involved in the international push for the assessment of research impact; the UK, US and Australia, which utilise substantially different assessment models. The project will develop a theoretical understanding of research impact and its measurement. It will identify the effects of national research impact assessment policies and practices, and ascertain how they shape the ways research is done and their consequences for knowledge.
Potential applications and benefits
There is a pressing need for in-depth studies of the institutional and individual effects of research evaluation to interrogate frequently uncritical acceptance of dominant forms of valuation. This study will be novel in its examination of the powerful rhetoric of value that dominates discussions of research impact, and in its articulation of the challenges involved in the audit and commodification of academic research. This research will advance sociological and political understandings of research value, and will contribute theoretically informed knowledge to the applied fields of research evaluation and public policy. It will examine the emerging cultural patterns that surround research impact, and shed light on the political processes of decision-making and authority around impact policies. The study will shape the academic debate on the impact cultures of the UK, US and Australia, and contribute to a larger debate on the value of publicly funded research.
The study has strong potential to influence the policies that determine how impact is measured and how publicly funded research is governed in the UK, US, Australia and globally. By working directly with stakeholders including science policymakers, research funders and researchers, it will inform debate on the role of universities and research in society and facilitate more robust, reflexive and ethical research systems. It will provide recommendations on the structures and resources that policymakers, funders and higher education leaders could make available to researchers to support them in achieving high-quality research that makes a meaningful contribution to society. Understanding research impact and its assessment is of critical importance, not only strategically for UK, US and Australian governments, but also for countries with similar or emerging evaluation and funding models. It will allow countries to be intentional about creating responsive, equitable research systems that promote core values, such as a dual focus on research excellence and societal benefit.
Outputs from the study will include dissemination to policymakers, practitioners and the wider public through events for stakeholders, a working paper, a policy brief, blogs and media articles and a project website. Academic dissemination will occur through conference participation and open access articles in top journals.
Organisations
Publications
Williams K
(2023)
Exploring the application of machine learning to expert evaluation of research impact.
in PloS one
Williams K
(2022)
Hybrid knowledge production and evaluation at the World Bank
in Policy and Society
Williams K
(2021)
Understanding, measuring, and encouraging public policy research impact
in Australian Journal of Public Administration
Williams K
(2022)
What counts: Making sense of metrics of research value
in Science and Public Policy
Williams K
(2023)
Investigating hybridity in artificial intelligence research
in Big Data & Society
Description | Cited in University of Melbourne Central Research Strategy |
Geographic Reach | Australia |
Policy Influence Type | Citation in other policy documents |
Impact | Academic work was the basis for the design of University of Melbourne's Chancellery strategy, which informs how the University invests in and manages research impact. |
Description | Participation in evaluation of the University Grants Committee (Hong Kong) Research Assessment Exercise |
Geographic Reach | Asia |
Policy Influence Type | Membership of a guideline committee |
Description | Participation in policy research project for the Australia Academy of Social Sciences, 'Modernising research assessment in Australia' |
Geographic Reach | Australia |
Policy Influence Type | Contribution to a national consultation/review |
Description | Participation in policy research project for the Royal Society, 'Taxonomy of Artificial Intelligence related technologies for scientific research' |
Geographic Reach | National |
Policy Influence Type | Contribution to a national consultation/review |
Description | National research impact policies: Uncovering the 'value' in evaluation' |
Amount | $164,764 (AUD) |
Organisation | Australian Research Council |
Sector | Public |
Country | Australia |
Start | 01/2024 |
End | 12/2027 |
Description | Wider societal value of research and consequences of its assessment: A multi-country and multi-method study (MultiSocVal) |
Amount | € 1,597,400 (EUR) |
Organisation | Volkswagen Foundation |
Sector | Charity/Non Profit |
Country | Germany |
Start | 08/2024 |
End | 08/2028 |
Description | Expert workshop on the theory and practice of research impact in international perspective |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other audiences |
Results and Impact | An invited group of 12 guests, involving both other scholars of research impact as well as impact/evaluation professional practitioners, attended a two-day workshop to discuss the project's interim findings and share related ongoing work/research. This resulted in the attendees successfully submitting a special issue proposal at the journal Research Evaluation (expected publication 2025), which has a reputation of reaching both research and practitioner audiences. |
Year(s) Of Engagement Activity | 2023 |
Description | Invited to UKRI-DSIT roundtable to discuss effective ways of supporting and funding R&D. |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Supporters |
Results and Impact | Invited to roundtable to contribute to discussion on how to best support and fund R&D, partly based on knowledge/expertise gained during this project. |
Year(s) Of Engagement Activity | 2024 |
Description | Presentation at Impactful Career Pathways Short Course, University of Melbourne |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Delivered talk to Impactful Career Pathways, drawing on Bourdieu's social theory as a framework to explore different forms of capital underpinning the five different facets of impact from scholarly and creative, to policy/government/law, to public debates, to practices and processes, and enterprise and commercialisation. |
Year(s) Of Engagement Activity | 2023 |
Description | Revisiting the role of research in society' public talk |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Public/other audiences |
Results and Impact | Convened a public panel discussion on research impact called 'Revisiting the role of research in society' at the University of Melbourne, on 9 September 2021. The live virtual event drew over 80 attendees, with many more watching the recording. |
Year(s) Of Engagement Activity | 2022 |
Description | Talk at Westminster Higher Education Forum |
Form Of Engagement Activity | A formal working group, expert panel or dialogue |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Policymakers/politicians |
Results and Impact | Talk at Westminster Higher Education Forum on research impact. |
Year(s) Of Engagement Activity | 2022 |
Description | Talk at the Innovation to Research Impact Assessment, Virtual International Convention for Research Administrators |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Talk given at the Innovation to Research Impact Assessment, Virtual International Convention for Research Administrators, 1-3 March 2022, a virtual international event run across four continents. |
Year(s) Of Engagement Activity | 2022 |