Valuing Different Perspectives - evaluation and evaluative knowledge

A central aim of the Connected Communities programme has been to carry out research with communities, not on communities. The AHRC are now keen to understand what the legacy of this investment in research with communities has been. The ambitious and complex nature of the resulting projects means there are significant challenges for attempts to evaluate legacy and whether they will be successful. The aim of this research will be to explore different approaches to evaluating projects and produce guidelines for future Connected Communities and similar projects. This will be carried out by delivering two evaluations of the same projects. One evaluation will be participatory, led by community partners; one will be academic-led.
Academic- or expert-led evaluations, such as those common in the social sciences, provide many ways to find out "what works" or provides value-for-money - randomised controlled trials; large scale surveys; cost-benefit analysis; theories of change; realist evaluation etc. Such evaluations may involve the use of quantitative or qualitative methods that are defined, created, and analysed by academic researchers, often regarded as experts, but who may be removed from the communities themselves which have been subject to a policy or project. Importantly, the evaluation criteria are also set by the academic research team, typically before the research project has begun. The evaluator has the power to define whether something has been a "success" or "failure" which can have a long-lasting impact.
In comparison, evaluations that are shaped and led by the community members at the heart of the projects are now increasingly used to assess the success and legacy of initiatives. These aim to capture the "local knowledge" of implementation and what was of value to the communities themselves. They respond to a concern that many of the softer outcomes of projects, that are often difficult to quantitatively measure, are not captured or even devalued by expert-led evaluations. Participatory evaluations aim to give communities the power to define "success" or "failure". This project, and the simultaneous evaluations at the heart of it, aims to explore these questions of power in the production of knowledge by asking what happens when community partners lead an evaluation and academics lead an evaluation of the same project. Are the research outputs produced different? And why might this be the case?
To answer these questions, the project will run workshops to supplement the evaluations. Attended by community members and academics, the first will discuss the processes, commonalities, differences, values and limitations of the various evaluation approaches chosen. The second workshop will develop a series of evaluation guidelines suited for future community-based research. Rather than being a "toolkit", this will be an exploration of the different methods used in the different contexts and what worked, or did not work, and why. This will enable future projects to make better informed decisions as to what evaluation methods they might use when planning community projects.
We cannot be prescriptive about the evaluation questions the community-led evaluation will ask. The academic evaluation will go back to original project briefs and align the original aims with the methods that the research team can use to ask whether the project has delivered what it set out to. The project also aims to explore the broader legacy of the Connected Communities programme. A key question in both evaluation sites will be the role of the AHRC as a funder of this type of activity and how this can be improved in future.

The proposal is designed to deliver direct impact for the communities involved. By running two evaluations we aim to deliver a broad range of information for the groups to practically use in assessing their activities and the impact of their own activities, supporting the future development of these, especially around funding bids.
A key aim for the project is to develop evaluation guidance outlining practical examples of applied evaluation methods in use in different contexts, assessing their practicality and usefulness. The guidelines will aid projects by improving practice and helping secure future funding. We foresee a number of indirect beneficiaries of this:
Community groups involved in existing, or developing Connected Communities projects, or other academic-community collaborations: the guidelines will provide a practical tool to empower these organisations to think about evaluation themselves and discuss different methods of evaluation and measurements of success or failure with academic partners at the outset of a project.
Community groups and third sectors organisations: the dissemination of the guidelines through wider networks will support a broader range of groups. We foresee these groups being able to use the knowledge of the different methods to inform their own evaluation activities and helping them choose the right methods or tools to use from the outset.
UK Government, devolved administrations and local government: a key policy thrust across the UK is to deliver services with communities through co-production or community empowerment, either as the "Big Society" or Scotland's Community Empowerment and Renewal Bill. A more indirect benefit may be delivered through research team contacts across government who can use it to inform their own assessment of community activities, and also help them support communities to evidence their impact.