Novel method for tracking the translation processes that lead to impact from Biomedical research - A pilot study

Lead Research Organisation: University of Sussex
Department Name: Sch of Business Management & Economics

Abstract

The MRC Economic Impact call states that "The MRC wishes to understand better the link between research and wider economic and societal impacts, and to use this understanding to improve strategies for the future support of research". To directly address this requirement, this one year pilot project will develop and test a new approach for evaluating biomedical research teams. The specific empirical focus is on researchers conducting Translational Research (TR) and our theoretical perspective applies recently developed concepts from economic geography to explore how TR may be aided by managing 'proximities' (closing gaps) between bodies of work, institutions or individuals. The project involves operationalising these concepts within a novel analytical framework and then testing it against three test cases. In each case, the unit of analysis will be a team previously supported by a programme grant.

This project's approach is novel because rather than attempting to measure research impacts, it will focus on the work processes used by TR scientists in the pursuit of impact. So rather than measuring impact, we aim to understand how research processes can be improved at a policy, strategy and practical level in order to enhance impact. Althought this is a pilot study, rather than just 1) testing the methods, and 2) streamlining them for future use, the project will also 3) provide outputs of direct relevance to funders and grant holders planning, funding and conducting effective TR.

This focus on processes also addresses a key methodological issue in impact assessment. Directly assessing impact using input-output methods is likely to be problematic because of the long time that is needed for advances in knowledge to be converted into widely adopted medical innovations, and the variety of actors and institutions involved (MRC 2012). An overly linear view of the process can be misleading. To avoid this problem, the project will use a mixed-method approach, involving both qualitative and quantitative components, to open the 'black box' of impact assessment and analyse the processes involved.

In so doing, this proposal addresses two of the objectives laid out in the call for proposals: Firstly, it will develop methodologies to track the pathways through which translational research generates socio-economic impacts; and secondly it will help identify success factors and develop partial intermediate indicators to assess progress towards research impact at the micro level (an individual project, centre, programme, or any specific investment). This project will deliver a broadly applicable method that allows for the timely evaluation of research investments by characterising the changes in TR work processes facilitated by specific grants.

The project team includes senior experts with specialist skills related to evaluation and the study of innovation in biomedicine. The team is led by Dr. Michael Hopkins, Director of Research and Senior Lecturer in SPRU, an expert in the study of biomedical Innovation and in the use of mixed methods studies. The team also includes Dr. Puay Tang, Director of Teaching and Senior Lecturer in SPRU and Prof. Jordi Molas-Gallart, of INGENIO, a Institute of the Spanish Council for Scientific Research and the Polytechnic University of Valencia, both with long experience in research evaluation work for UK an EU funders. The team also draws on the expertise of Dr. Ismael Rafols, a Visiting Fellow at SPRU and Research Fellow at INGENIO, and internationally leading expert in bibliometrics, as well as Prof. Gail Davey and Prof. Tony Carr, who bring years of experience working on laboratory and clinical research projects to the team.

Technical Summary

Evaluations of research impact in the biomedical sciences often focus on measuring inputs and outputs with little emphasis on understanding and tracing the actual processes that lead to impact. Input-output studies can be problematic because of the long time needed for valued impacts to develop, the large number of actors and institutions involved in a given advance, and an overly linear view of causation which can be misleading (MRC 2012). To avoid such problems, this pilot project will instead focus on understanding and tracing changes in the work processes that researchers use when striving for research impact. This alternative method will be developed and tested by following the progress of eight cases, each of which will comprise a team of researchers conducting Translational Research (TR) following the awarding of a grant.
Our method is based on a theoretical perspective developed from economic geography and applied to TR, where we conceptualise translation as a change in 'proximities' (closing gaps) between bodies of work, institutions or individuals which can be detected through tracing changes in work processes. The method relies on both qualitative interview and quantitative bibliometric techniques to conduct evaluation.
This project's methods approach is novel because it brings into the field of research evaluation an analytical framework initially developed in economic geography. We will thus use together quantitative techniques from bibliometrics used in economics of innovation (measuring proximities between topics, actors, organisation, etc.), with semi-structured interviews. Linking these two methodologies is a contribution on its own.
The project team comprises of four scholars with strong track records in research evaluation and the study of biomedical innovation and building on a strong tradition of such work at SPRU, University of Sussex.

Planned Impact

This 18 month pilot project attempts to test a new approach to the evaluation of biomedical research. Yet it is important even at these early stages of development, to involve stakeholders in the research. Through this direct involvement we will increase the likelihood of adoption and ultimately impact. We identify three main types of direct stakeholders: biomedical researchers, funding agencies, and evaluation practitioners in academia and consultancies.

Biomedical researchers will be directly involved in the project. We do not see them as passive subjects, but as parties interested in developing evaluation tools that can help them improve the efficiency and efficacy of their research. Collaboration with biomedical researchers is a crucial aspect of our approach to methodology development. Not only will we use the interview and questionnaire protocols to seek explicitly information about their perspectives on the implementation of evaluation tools, but we will keep close contact with the projects PIs throughout the study. In addition, we will conduct one workshop in every centre to present interim results and engage the researchers in the case study summary that will constitute the main deliverable of this pilot.

Second, we expect to offer research funding organisations a new set of evaluation tools. We will focus on the MRC, conducting the study in dialogue with MRC officers - who are in an excellent position to learn from the findings and contribute to their interpretation. MRC officials will be consulted at each of the three phases of the project and we plan to conduct a workshop with MRC and other funding organisations (Wellcome Trust, CR-UK) before the drafting of the pilot's final briefing report on the use of our method.

Third, the engagement of evaluation practitioners will be conducted through conventional dissemination means. We will aim to disseminate our approach through:

(1) Writing features on the project in practitioner blogs (recently Rafols produced a blog for the US National Cancer Institute's on interdisciplinary practices, and Tang and Molas-Gallart produced a blog for the "LSE Impact Blog"). Rafols is also actively involved in the running of the website www.interdisciplinaryscience.net where analysis and evaluation methods are freely shared and discussed and Hopkins written blogs for Research Fortnight.

(2) Through publication of results in the academic, practitioner-oriented evaluation and medical journals for example Research Evaluation or Nature Biotechnology respectively.

(3) Through our participation in international fora concerned with evaluation practice. The team is engaged with several international organisations. For example, Rafols is engaged in advisory meetings for the ERC and OECD on S&T indicators, and Molas-Gallart is a member Science Europe Working Group on Ex-Post Evaluation.

In addition we are in a position to implement the approaches developed in our evaluation practice. The involvement in the project of two international research institutes based in two different countries (SPRU and INGENIO) with their own networks of collaborators provides a very wide scope for future development and implementation of the approaches the pilot will develop. For example, only over the last two years, Hopkins carried out a consultancy for NESTA and CR-UK, Tang and Molas-Gallart two for the ESRC and Molas-Gallart participated on a project on best practice on the delivery of care in rare diseases for the EU Joint Action EUCERD. In addition Rafols has provided training on mapping methods to private companies such as Technopolis (UK) and Search Technology (US). We expect this type of activities to continue in the near future and to use them as a springboard to implement and further develop the approaches developed in this pilot.

Publications

10 25 50
 
Description Research on Research
Amount £169,690 (GBP)
Organisation Wellcome Trust 
Sector Charity/Non Profit
Country United Kingdom
Start 03/2019 
End 02/2021
 
Description Collaboration on report : the value of structural diversity 
Organisation Digital Science
Country United Kingdom 
Sector Private 
PI Contribution Staff at SPRU, including those developping the DARE methodology, have published a series of papers and studies on diversity in academic research. Digital Science approached SPRU to produce a joint report that synthesised the key messages from SPRU's prior research together with other research undertaken at Digital Science and other organisations. The DARE team supported Digital Science who led on the report. Michael Hopkins and Ismael Rafols were involved in discussions with Digital Science and led on coordinating with the Digital Science team.
Collaborator Contribution Digital Science has long been aware of leading research at SPRU on diversity. Digital Science took the initiative to approach SPRU and initiate the collaboration to produce an accessible report for the research evaluation community. Digital Science met the full cost of producing the report, took the lead in producing the report and made i available in open access format online.
Impact Report on ' The value of structural diversity' (https://s3-eu-west-1.amazonaws.com/pstorage-digitalscience-874864551/3625932/DRR4Dec151.pdf)
Start Year 2015
 
Description Researchfish 
Organisation Researchfish
Country United Kingdom 
Sector Private 
PI Contribution The DARE team is exploring ways to use the "Diversity Approach to Research Evaluation" to enhance the application Researchfish.
Collaborator Contribution Researchfish provides the DARE team with access to practical experience in the use of research evaluation tools and systems as well as valuable insights into the context for their use in the UK and internationally. Researchfish also provides support to the DARE team that helps the team to reach an audience of research managers and funders.
Impact The collaboration has supported the DARE team in applying successfully for a Wellcome Trust grant.
Start Year 2017
 
Title Software routines for data visualisation associated with the Diversity Approach to Research Evaluation 
Description Software routines have been developed to support the generation of measures and maps that can describe team-based research collaborations. Examples of the application of these routines can be seen here: http://www.sussex.ac.uk/spru/research/dare/Cases.html 
Type Of Technology Software 
Year Produced 2017 
Impact None as yet. 
URL http://www.sussex.ac.uk/spru/research/dare/Cases.html
 
Description Meeting with Fiocruz (Brazil) research evaluation team 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact FioCruz is Brazil's largest funder of biomedical research. Members of the FioCruz research evaluation team and SPRU have discussed a potential collaboration between the two intstitutions focused on bibliographic analysis (including the indicators developped in the grant). Fiocruz and Sussex are developing a memorandum of understanding and exploring opportunities for collaborative funding.
Year(s) Of Engagement Activity 2015
 
Description Meeting with GSK to discuss methods to measure their research outputs and see team activities using DARE 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Industry/Business
Results and Impact This was a small meeting between two GSK representatives and 4 researchers at the University of Sussex. We discussed the various methods we are developing to understand how teams collaborate and how to measure outputs. After the meeting we exchanged emails about further potential collaboration.
Year(s) Of Engagement Activity 2018
 
Description Meeting with NIHR research evaluation group in Southampton 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact About 10 people from the NIHR Southampton group on Research evaluation attended a presentation on DARE. We discussed the novelty of the method developed within our project and discussed the aspects of the methods that could be useful for research evaluation. The group reported that the method was of interest to them, and would discuss it's potential use in a further meeting of their group.
Year(s) Of Engagement Activity 2018
 
Description Participation in Researchfish User Advisory Group 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Michael Hopkins (project PI) was invited to join a small group of 5-10 Researchfish (RF) users in order to feed into a major redesign following recognition by RF that improvements to their service would improve academic engagement with impact assessment. Over the course of several meetings at RF and Sussex, MH contributed suggestions for the RF redesign, drawing on his own user experience and also experience gained from research on tracking translational research projects as part of this MRC funded research.

Although it is early days, first signs are that the RF redesign has been very successful with users suggesting that the site has become much more user-friendly.
Year(s) Of Engagement Activity 2016
 
Description Presentation at a policy/practitionner conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation at "New Data for Innovation Policy" a OECD /EU commission / Nesta workshop in London to about 50 practitioners, policy makers and funders to discuss the visualisation of the method and how it relates to the concept of diversity.
Year(s) Of Engagement Activity 2019
 
Description Presentation at a policy/practitionner conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Presentation at the IGL conference 2019 in Berlin to about 50-60 people about the novel interactive visualisation methods implemented in DARE.
Year(s) Of Engagement Activity 2019
URL https://www.innovationgrowthlab.org/igl2019-global-conference
 
Description Presentation at the Researchfish advisory board 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact This meeting introduced DARE to research funders and also was seeking support for an application for further funding. This led to a further invitation to attend the Researchfish conference, and further dissemination of the method. Collaboration for a grant application was agreed (which was successful in leading to further funding).
Year(s) Of Engagement Activity 2018
 
Description Presentation at the Researchfish annual conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We presented the DARE method to an audience of research funders, professional services and researchers in research evaluation, to an audience of 30-40 people. Many funders were encouraging of the methods and interested in its future development and potential to be used in the understanding of their own research portfolio.
Year(s) Of Engagement Activity 2018
 
Description Research on Research meeting at the Wellcome Trust 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact This was a workshop to discuss between funders and researchers possible further research opportunities in the area of "Research on Research" in the upcoming years, to be relevant for the funders needs.
Year(s) Of Engagement Activity 2018
 
Description Workshop introducing the Diversity Approach for Research Evaluation (DARE) 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact 30 policy practitioners, funders, and academics attended this workshop which aimed at presenting the novel method developed during the course of the project. It introduced the aim, objective and practical application of the new method. It also presented a draft version of a user guide together with a website aimed at dissemination.
Year(s) Of Engagement Activity 2016
 
Description Workshop to gain stakeholder feedback 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Third sector organisations
Results and Impact The project team presented the DARE approach to an audience of research evaluation professionals, funders and research managers at a joint meeting hosted by the National Institute for Health Research and the Association of Medical Research Charities (in London), in order to gain feedback from them on how DARE could be made more applicable for evaluating scientific collaboration in their area. Around 30 individuals attended the event.
Year(s) Of Engagement Activity 2019
 
Description Workshop to gain stakeholder feedback 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The project team presented the DARE approach to an audience of researchers and research managers at the Francis Crick Institute (London), in order to gain feedback from them on how DARE could be made more applicable for evaluating scientific collaboration in their area. Around 20 individuals attended the event.
Year(s) Of Engagement Activity 2019
 
Description Workshop to gain stakeholder feedback 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The project team presented the DARE approach to an audience of research evaluation professionals UKRI (Swindon), in order to gain feedback from them on how DARE could be made more applicable for evaluating scientific collaboration in their area. Around 20 individuals attended 2 events (in July 2019 and January 2020).
Year(s) Of Engagement Activity 2020
 
Description Workshop to gain stakeholder feedback 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The project team presented the DARE approach to an audience of researchers and research managers at the Oxford Biomedical Research Centre, in order to gain feedback from them on how DARE could be made more applicable for evaluating scientific collaboration in their area. Around 10 individuals attended the event.
Year(s) Of Engagement Activity 2020
 
Description Workshop to gain stakeholder feedback 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The project team presented the DARE approach in poster format at the Researchfish annual conference (London), in order to gain feedback from them on how DARE could be made more applicable for evaluating scientific collaboration in their area. The team distributed a large number of research briefs detailing DARE to conference delegates.
Year(s) Of Engagement Activity 2019