Understanding the "impact" of health research: Lessons from the REF2014

In 1976, a German researcher linked sun bed use to skin cancer. In 2006, 30 years later, the Cooksey review of publicly funded health research stated that the UK was at risk of "failing to reap the full economic, health and social benefits of public investment in health research" due to research not being adequately translated into health outcomes. That same year, a 26 year old girl, Clare Oliver, was diagnosed with advanced melanoma and devoted her last 3 months to highlighting the dangers of the solarium industry, which she blamed for her melanoma. Clare's personal media campaign triggered state and federal health ministers to regulate the solarium industry and restrict its use by people with melanoma-prone skin. By the time Clare passed away, she had left behind a public health policy that had taken researchers over 30 years to achieve without her. Claire's story demonstrates Cooksey's concern: why did the strong research linking melanoma to sun bed use, first published in 1976, take so long to inform this health policy? The answer lies in current problems with translating high quality health research into social outcomes (societal impact).
In 2014, a new evaluation framework, the Research Excellence Framework (REF2014) will use a measure of how research has influenced the social, political and economic lives of citizens in order to distribute almost £1.76 billion of public funding. Including this new criterion, however, raises questions about how evaluations of societal impact will be made and what aspects are evaluated favourably. For health research, the societal benefits may be obvious, but assessing them is difficult.
In Clare's case, how do we reward the research involved? Identifying which research was most responsible is challenging because it is difficult to pinpoint which research was the most influential. Was it the research that first linked solarium use to an increased risk of melanoma in 1976? Should it be Clare's doctor, the author of the systematic review, the researcher that sat on the government committee, or the person who drafted the policy? The incorporation of societal impact in the REF2014 evaluations provides a unique opportunity to investigate how research evaluators handle this new impact criterion.
The academic literature has yet to provide answers to these public health- and evaluation-policy issues, and several other countries (Australia, The Netherlands, New Zealand and The US) wish to include societal impact criteria in their evaluation policies, but are yet to be successful. A number of models to guide evaluations have been proposed, including those that solely measure the societal outcome (in Clare's case, the solarium policy) and those that focus on researcher behaviours known to be influential to achieve a societal impact (media engagement or advocacy). Unfortunately, the academic literature is undecided on which aspects are best used to guide evaluations.
My research will investigate how evaluators define societal impact during the REF2014 evaluation. This timely research topic will use both quantitative and qualitative methodologies, including analyses of how government policy documents refer to the concept of societal impact, the evaluator's academic background, and interviews with evaluators before and after the REF2014 evaluation process. The research will also investigate how evaluators use their own experience of societal impact to inform their how they define societal impact as part of research excellence.
The results of my research will influence the development of evaluation policies that incorporate societal impact considerations both in the UK and abroad. It will also produce high-quality academic research articles that will be used to inform research on the importance of societal impact, contributing to a realisation that a definition of excellent research also includes how health research affects the quality of our everyday lives.

My proposed research investigates how the concept of societal impact is evaluated for health research. The incorporation of the formal "impact" criterion for REF2014 provides the perfect opportunity for research that can inform relevant policies addressing the concerns of Cooksey (2006) that society is failing to sufficiently benefit from its investment in health research.
The following stakeholder groups have been identified as beneficiaries of this research.
(1) Funding bodies, research managers and government policymakers keen to develop research evaluation policies that incorporate and therefore incentivise societal impact activities.
The proposed research will help guide evaluation policies for organisations such as the Higher Education Funding Council for England (HEFCE) and the individual funding councils, including the ESRC and MRC. My own experience combined with the experience of Dr Donovan working with government organisations will ensure that the research results are used effectively to develop relevant evaluation guidelines and policies. Ultimately, however, the main beneficiary within this stakeholder category is future Research Excellence Frameworks run by HEFCE. As the REF2014 represents the first framework with a formal societal impact component, this research will provide information on the success of this endeavour and provide feedback to improve its evaluation in the future.
Outside the UK, this research will benefit a number of countries (Australia, New Zealand, The Netherlands, USA) and organisations (European Commission and the National Science Foundation) that have expressed interest in incorporating societal impact components of evaluation policies but have yet to do so successfully. Dr Donovan's experience in advising governments for developing evaluation frameworks (RQF and ERA in Australia and HEFCE for REF2014) will help to facilitate this impact.
(2) Researchers, universities and university research offices that require guidance on how to sufficiently prepare submissions for evaluation processes that incorporate a societal impact criterion.
Currently, UK universities are preparing 1000 word case studies about how their research has achieved a societal impact. This is being done without a clear understanding of how each impact summary will be valued by evaluators. The academic articles produced by this proposed research will increase awareness about what aspects of health research translation are valued by research evaluators. This will not only help universities and research organisations to incentivise and encourage staff to engage with the societal impact of research, but it will also help them prepare stronger submissions for future frameworks that incorporate a societal impact criterion.
(3) Research evaluators who require guidelines on how to equitably reward societal impact activities.
An important beneficiary of this research will be evaluators who are being asked to use the societal impact of the research in addition to traditional, academic criteria to make evaluations. Evaluating the societal impact of research is a new concept and as such, evaluators may be ill-prepared to evaluate these criteria effectively. The proposed research aims to influence the development of policies to help evaluators confidently evaluate this criterion.
In addition, by influencing evaluators, my research will ensure that an understanding of excellent societal impact is transmitted to more local-level evaluation processes such as university appointments, promotions and grant-evaluation committees.
Finally, by understanding how the societal impact of research is valued, researchers will have a clear incentive to engage with activities that maximise the socio-economic benefit of their research. By incentivising this aspect of research, the general public is positioned to benefit indirectly from my proposed research plan as well.