Novel method for tracking the translation processes that lead to impact from Biomedical research - A pilot study

The MRC Economic Impact call states that "The MRC wishes to understand better the link between research and wider economic and societal impacts, and to use this understanding to improve strategies for the future support of research". To directly address this requirement, this one year pilot project will develop and test a new approach for evaluating biomedical research teams. The specific empirical focus is on researchers conducting Translational Research (TR) and our theoretical perspective applies recently developed concepts from economic geography to explore how TR may be aided by managing 'proximities' (closing gaps) between bodies of work, institutions or individuals. The project involves operationalising these concepts within a novel analytical framework and then testing it against three test cases. In each case, the unit of analysis will be a team previously supported by a programme grant.

This project's approach is novel because rather than attempting to measure research impacts, it will focus on the work processes used by TR scientists in the pursuit of impact. So rather than measuring impact, we aim to understand how research processes can be improved at a policy, strategy and practical level in order to enhance impact. Althought this is a pilot study, rather than just 1) testing the methods, and 2) streamlining them for future use, the project will also 3) provide outputs of direct relevance to funders and grant holders planning, funding and conducting effective TR.

This focus on processes also addresses a key methodological issue in impact assessment. Directly assessing impact using input-output methods is likely to be problematic because of the long time that is needed for advances in knowledge to be converted into widely adopted medical innovations, and the variety of actors and institutions involved (MRC 2012). An overly linear view of the process can be misleading. To avoid this problem, the project will use a mixed-method approach, involving both qualitative and quantitative components, to open the 'black box' of impact assessment and analyse the processes involved.

In so doing, this proposal addresses two of the objectives laid out in the call for proposals: Firstly, it will develop methodologies to track the pathways through which translational research generates socio-economic impacts; and secondly it will help identify success factors and develop partial intermediate indicators to assess progress towards research impact at the micro level (an individual project, centre, programme, or any specific investment). This project will deliver a broadly applicable method that allows for the timely evaluation of research investments by characterising the changes in TR work processes facilitated by specific grants.

The project team includes senior experts with specialist skills related to evaluation and the study of innovation in biomedicine. The team is led by Dr. Michael Hopkins, Director of Research and Senior Lecturer in SPRU, an expert in the study of biomedical Innovation and in the use of mixed methods studies. The team also includes Dr. Puay Tang, Director of Teaching and Senior Lecturer in SPRU and Prof. Jordi Molas-Gallart, of INGENIO, a Institute of the Spanish Council for Scientific Research and the Polytechnic University of Valencia, both with long experience in research evaluation work for UK an EU funders. The team also draws on the expertise of Dr. Ismael Rafols, a Visiting Fellow at SPRU and Research Fellow at INGENIO, and internationally leading expert in bibliometrics, as well as Prof. Gail Davey and Prof. Tony Carr, who bring years of experience working on laboratory and clinical research projects to the team.

Evaluations of research impact in the biomedical sciences often focus on measuring inputs and outputs with little emphasis on understanding and tracing the actual processes that lead to impact. Input-output studies can be problematic because of the long time needed for valued impacts to develop, the large number of actors and institutions involved in a given advance, and an overly linear view of causation which can be misleading (MRC 2012). To avoid such problems, this pilot project will instead focus on understanding and tracing changes in the work processes that researchers use when striving for research impact. This alternative method will be developed and tested by following the progress of eight cases, each of which will comprise a team of researchers conducting Translational Research (TR) following the awarding of a grant.
Our method is based on a theoretical perspective developed from economic geography and applied to TR, where we conceptualise translation as a change in 'proximities' (closing gaps) between bodies of work, institutions or individuals which can be detected through tracing changes in work processes. The method relies on both qualitative interview and quantitative bibliometric techniques to conduct evaluation.
This project's methods approach is novel because it brings into the field of research evaluation an analytical framework initially developed in economic geography. We will thus use together quantitative techniques from bibliometrics used in economics of innovation (measuring proximities between topics, actors, organisation, etc.), with semi-structured interviews. Linking these two methodologies is a contribution on its own.
The project team comprises of four scholars with strong track records in research evaluation and the study of biomedical innovation and building on a strong tradition of such work at SPRU, University of Sussex.

This 18 month pilot project attempts to test a new approach to the evaluation of biomedical research. Yet it is important even at these early stages of development, to involve stakeholders in the research. Through this direct involvement we will increase the likelihood of adoption and ultimately impact. We identify three main types of direct stakeholders: biomedical researchers, funding agencies, and evaluation practitioners in academia and consultancies.

Biomedical researchers will be directly involved in the project. We do not see them as passive subjects, but as parties interested in developing evaluation tools that can help them improve the efficiency and efficacy of their research. Collaboration with biomedical researchers is a crucial aspect of our approach to methodology development. Not only will we use the interview and questionnaire protocols to seek explicitly information about their perspectives on the implementation of evaluation tools, but we will keep close contact with the projects PIs throughout the study. In addition, we will conduct one workshop in every centre to present interim results and engage the researchers in the case study summary that will constitute the main deliverable of this pilot.

Second, we expect to offer research funding organisations a new set of evaluation tools. We will focus on the MRC, conducting the study in dialogue with MRC officers - who are in an excellent position to learn from the findings and contribute to their interpretation. MRC officials will be consulted at each of the three phases of the project and we plan to conduct a workshop with MRC and other funding organisations (Wellcome Trust, CR-UK) before the drafting of the pilot's final briefing report on the use of our method.

Third, the engagement of evaluation practitioners will be conducted through conventional dissemination means. We will aim to disseminate our approach through:

(1) Writing features on the project in practitioner blogs (recently Rafols produced a blog for the US National Cancer Institute's on interdisciplinary practices, and Tang and Molas-Gallart produced a blog for the "LSE Impact Blog"). Rafols is also actively involved in the running of the website www.interdisciplinaryscience.net where analysis and evaluation methods are freely shared and discussed and Hopkins written blogs for Research Fortnight.

(2) Through publication of results in the academic, practitioner-oriented evaluation and medical journals for example Research Evaluation or Nature Biotechnology respectively.

(3) Through our participation in international fora concerned with evaluation practice. The team is engaged with several international organisations. For example, Rafols is engaged in advisory meetings for the ERC and OECD on S&T indicators, and Molas-Gallart is a member Science Europe Working Group on Ex-Post Evaluation.

In addition we are in a position to implement the approaches developed in our evaluation practice. The involvement in the project of two international research institutes based in two different countries (SPRU and INGENIO) with their own networks of collaborators provides a very wide scope for future development and implementation of the approaches the pilot will develop. For example, only over the last two years, Hopkins carried out a consultancy for NESTA and CR-UK, Tang and Molas-Gallart two for the ESRC and Molas-Gallart participated on a project on best practice on the delivery of care in rare diseases for the EU Joint Action EUCERD. In addition Rafols has provided training on mapping methods to private companies such as Technopolis (UK) and Search Technology (US). We expect this type of activities to continue in the near future and to use them as a springboard to implement and further develop the approaches developed in this pilot.