Choices of Evidence. Tacit philosophical assumptions in the debates within the Campbell Collaboration

Evidence-based policy is now mandated throughout the UK and the USA, and increasingly in Europe, at international, national and local levels. Modelled on the Cochrane Collaboration in medicine, the Campbell Collaboration is an independent, international, non-profit organization which has been set up to provide 'decision-makers with evidence-based information to empower them to make well-informed decisions about the effects of interventions in the social, behavioural and educational arenas' (www.campbellcollaboration.org). There are, however, major debates, both within the Campbell Collaboration and with its critics, about what counts as acceptable evidence in evidence-based policy and practice and how its strength as support for a particular hypothesis can be judged.
This project, focusing on the cases studies of child social welfare and education, aims to clarify and help resolve these controversies by analysing the philosophical assumptions that tacitly inform them. Most of the controversies appear to originate from different demands put on what evidence should deliver and what it can deliver, given the level of the knowledge base and the complexity of the subject matter.
We shall examine a number of well-established and yet disputable distinctions that we conjecture add confusion to the debate. It often seems that opponents are failing to understand each other's positions because they are implicitly adopting different philosophical assumptions about the nature of the social world or of scientific method.

We shall address six key issues. First, quantitative and qualitative data: a distinction between the two is widely accepted with the former being generally considered more valid and reliable than the latter. But in social welfare and education, practitioners reason on the basis of partial and ambiguous data about people's behaviour. In what ways precisely do quantitative and qualitative data differ and in what ways is the former purportedly more reliable.

A linked issue is the distinction between objective and subjective that, in practice, becomes a distinction between general and local knowledge, namely between methodology/expertise and more judgment-based approaches . Can the advantages of the general knowledge and expertise in research methods be gained without sacrificing local knowledge?

Third, the area of social welfare raises questions about who decides what counts as a solution in social interventions and how competing claims should be handled/adjudicated. In a subject area that deals with the least powerful groups in society, this issue is of special importance.

A fourth issue is whether a clear hierarchy of methods inadvertently leads to a hierarchy of questions, inevitably prioritising those questions that can be studied by the higher ranked methods.

The fifth question addresses the difference between 'clinching' methods of evidence (their conclusions are deduced from a positive result) and ''vouching' ones (conclusions upheld by evidence deemed to be satisfactory, possibly relevant etc. Clinching evidence requires so many assumptions and conditions that its application becomes, we fear, very limited. How should the trade-off between scope of application and certainty of conclusion be managed?

The final issue questions the difference between closed and open research methods. The former, such as RCTs, have built-in devices for checking the assumptions that are needed to apply the method. The latter, such as Bayes-net,require appeal to other devices. Need best evidence be supplied by closed methods or can a wider range be acceptable?

The outcome of these analyses will provide the basis for articulating a philosophy of science that meets the requirements and needs of practical research. We will aim at suggesting a theory of evidence that is both theoretically sound and realistically enforceable and thus contributes to the successful development of the Campbell project.

Not Required