Serendipity in Science, Technology and Innovation

I present the Serendipity proposal, with SPRU (Sussex, UK) and Columbia (NYC, USA), to conduct a body of frontier work on the role, management and impact of serendipity on research performance. The project addresses major theoretical issues (related to the governance of research) that are of especially significant policy importance to the UK as it emerges from a major recession and moves towards a more knowledge intensive economy.

The key focus of the project is on the under-researched topic of "serendipity", the notion in science policy that basic research leads to unexpected valuable outcomes, and since the outcomes of basic research are impossible to predict, research itself is difficult (perhaps even impossible) to manage or direct towards specific social ends. Serendipity has been a core idea in science, technology and innovation policy but there has been almost no research (with the exception of work by Merton, see later in Case for Support section) that has explored its relative importance, measured its magnitude or investigated its factors, and how it leads to social and economic benefits.

This is an important issue for the UK, given its relative innovation performance compared to Germany, USA and Asia. On the one hand, UK innovation policy is seen as being more blue-sky orientated than the USA (where the majority of publicly funded research is mission-orientated), yet on the other hand it is seen as missing (serendipitous) opportunities to exploit and 'translate' its research strengths into medical industries and patient benefits (Cooksey 2006). Pointing out that uncertainty means research cannot be micro-managed misses the more pertinent issue about whether UK science can be better managed in ways that enhance the social and economic value of serendipity for the UK. Research may be uncertain, but it is not random, and we know that industrial R&D managers fund research in areas where they expect returns and organise research to maximise its impact. With public policy, the scenario is slightly different, but there is not yet a body of evidence to draw on to support policy making.

To address this research gap, this project brings together two of the leading research groups in the world working on science policy. The project will be led by myself (a junior/early career researcher with a double prize-winning publication record), mentored by Prof Ben Martin, (former Director of SPRU), and Dr Bhaven Sampat, (tenured Associate Professor at Columbia). The project will undertake a mix of fundamental basic research on the nature and history of serendipity, and its influence on research policy, together with applied policy-focused research on issues of direct relevance to government policy makers, medical charities and industrial R&D managers. It will examine data and rare archives in the U.S. (that are accessible only on-location), with cutting edge quantitative and qualitative methods to generate large scale evidence as well six detailed cases studies. It will then focus on developing theory and implications to inform future UK policy on research governance.

A vital debate exists about the benefits of untargeted, fundamental, 'blue skies' research led by 'pure' enquiry relative to mission-oriented, applied research targeted at specific goals (Sarewitz 1996; Brooks 1995). It directly informs how science and technology should tackle prominent and pressing challenges, including climate change, food security and health-burdens, such as by developing energy-efficient technologies, better crops, and new drugs and medical devices (Mowery, Nelson and Martin 2010). The notion that unexpected benefits arise from basic research has been a centrepiece of U.K., European and U.S. policy, as well as underpinning interdisciplinary theories of innovation and social change.

Serendipity was a pillar of Vannevar Bush's 'blueprint' for post-war U.S. science and technology policy: "Scientific progress on a broad front results from the free play of free intellects, working on subjects of their own choice, in the manner dictated by their curiosity for exploration of the unknown." (Bush 1945, p12; Nelson 1997; Mowery 1997). The UK's Medical Research Council, which responds to curiosity-driven, investigator-initiated proposals, has a foreword in its 2009-2014 Strategy: "We must remain driven by the needs of populations, individuals and patients and the desire of scientists to strive for new discoveries... by supporting basic science. This not only delivers discoveries but provides the basis for translation of research into patient benefit." (MRC 2009-2014; p4). Basic researchers use the serendipity notion to promise utility on the one hand and justify autonomy on the other. In the U.K., the Haldane Principle is used to claim that research funding decisions should be made by researchers rather than politicians (Haldane 1918; Edgerton 1996).

Serendipity has persisted in U.K. policy debate despite notable counter-ideas, such as the customer-contractor principle that argues that government should behave as customer and scientists as contractor because, no matter how intelligent or practical, scientists cannot decide society's needs and priorities (Rothschild 1971). By 2008, U.K. policy had re-emphasised "respecting the spirit of Haldane" saying that a) "researchers are best placed to determine detailed priorities", adding that b) "the government's role is to set the over-arching strategy" (Denham 2008).

The relationship between 'detailed priorities' and 'over-arching strategy' remains a difficult, poorly-defined challenge for policymakers, with a sense that somehow serendipity would luckily lurk in the background. These arguments resemble, though do not mimic, contemporary debates about whether industrial policy should aim to affect not only the rate, but also the direction of scientific and technical change (Nelson 1997; Stirling 2009; Cimoli, Dosi, and Stiglitz 2009). Hence, they are relevant for other practitioners too, such as R&D managers in UK pharmaceutical firms like GSK, and major UK medical charities such as the Wellcome Trust, Leverhulme Trust and Cancer Research UK.