Categorisation of citations to trace the translation and impact of biomedical research

The importance of assessing the wider impact of biomedical research in terms of improved health is increasingly being recognised by research funding bodies. Such studies are used to account to tax payers for the money spent. They are also used, for example by medical research charities such as the Arthritis Research Campaign, in an attempt to understand how to organise research so as to enhance the benefits and thus lead to further improvements in health. But there are considerable practical problems in conducting such studies. Traditional approaches, involving researchers considering the work of their colleagues, play an important role in the evaluation of research quality. But they often play a more limited part in assessment of impacts such as how research is eventually translated into improved clinical practice. Considerable progress is made using newer approaches that involve conducting interviews with researchers and others to assess wider impacts of particular pieces of biomedical research. These studies are, however, resource-intensive to conduct and tend to be even more difficult to undertake in relation to basic biomedical research conducted in laboratories than on clinical research conducted in hospitals or the community.

A recent report recommended further work to develop evaluation methods. One approach sometimes used is to count the number of times other researchers refer to, or cite, a researcher?s article in their own research. This is known as citation analysis. How far such simple citation counts provide adequate measures of research quality is contested, but it is widely agreed that such counts do not provide adequate assessments of wider impacts from research.

We propose, however, to examine how a new approach to citation analysis could be used in tracing the impact of biomedical research and in illuminating the processes involved in the translation of research into eventual impact on clinical practice. To do this we propose a two phase approach. First, we propose to develop a method by which we shall be able to categorise citations and identify those that show that previous research has been important to a subsequent piece of research. Second, we propose to pilot the use of that method to trace the impact made by an original piece of research on a series of later pieces of research. By tracing the impact through such a series of papers we hope we might show how the original research eventually contributed to improved clinical care and improvements in health.

The importance of assessing the impact and translation of biomedical research is increasingly being recognised by research funding bodies and in reports such as those by Cooksey and the Evaluation Forum. There are, however, considerable methodological and practical problems in assessing impact of research. Traditional peer-review approaches play an important role in the evaluation of research, but often play a more limited part in assessment of impacts and can be costly. Considerable progress is made using case studies - often organised using the Payback Framework developed by Buxton and Hanney at HERG - to assess wider impacts of biomedical research. Such studies are used not only for accountability purposes but also by medical research charities in an attempt to organise research so as to enhance the benefits and thus lead to improved health. Such studies are, however, resource-intensive to conduct and tend to be even more difficult to progress in relation to basic than clinical research. One attempt to reduce the cost of conducting impact studies is to see how far the results of surveys of PIs are as valid as more resource-intensive case studies. But much of the current focus of surveys is on more recently completed research.

The Evaluation Forum recommended further work to develop evaluation methods. Whether some new forms of citation analysis can contribute to the assessment of long-term outcomes from research is an important but under-explored issue. Citation analysis methods currently are almost invariably simple quantitative techniques and, therefore, more mechanised and less resource-intensive than other approaches. How far such simple citation counts provide adequate measures of research quality is contested, but it is widely agreed that such counts do not provide adequate assessments of wider impacts from research.

We propose to examine how qualitative citation analysis could be used in tracing the impact of biomedical research and in illuminating the processes involved in the translation of research into eventual impact on clinical practice. To do this we propose a two phase approach. First, we propose to develop a citation categorisation template and test it as a method for application to biomedical research to assess the contribution that individual papers make to further research. Second, we propose to pilot the use of the template as a method that could be used as part of a study of streams of research aimed at tracing their translation and impact through a series of papers.