Valuing Different Perspectives - evaluation and evaluative knowledge

Lead Research Organisation: University of Stirling
Department Name: Applied Social Science

Abstract

A central aim of the Connected Communities programme has been to carry out research with communities, not on communities. The AHRC are now keen to understand what the legacy of this investment in research with communities has been. The ambitious and complex nature of the resulting projects means there are significant challenges for attempts to evaluate legacy and whether they will be successful. The aim of this research will be to explore different approaches to evaluating projects and produce guidelines for future Connected Communities and similar projects. This will be carried out by delivering two evaluations of the same projects. One evaluation will be participatory, led by community partners; one will be academic-led.
Academic- or expert-led evaluations, such as those common in the social sciences, provide many ways to find out "what works" or provides value-for-money - randomised controlled trials; large scale surveys; cost-benefit analysis; theories of change; realist evaluation etc. Such evaluations may involve the use of quantitative or qualitative methods that are defined, created, and analysed by academic researchers, often regarded as experts, but who may be removed from the communities themselves which have been subject to a policy or project. Importantly, the evaluation criteria are also set by the academic research team, typically before the research project has begun. The evaluator has the power to define whether something has been a "success" or "failure" which can have a long-lasting impact.
In comparison, evaluations that are shaped and led by the community members at the heart of the projects are now increasingly used to assess the success and legacy of initiatives. These aim to capture the "local knowledge" of implementation and what was of value to the communities themselves. They respond to a concern that many of the softer outcomes of projects, that are often difficult to quantitatively measure, are not captured or even devalued by expert-led evaluations. Participatory evaluations aim to give communities the power to define "success" or "failure". This project, and the simultaneous evaluations at the heart of it, aims to explore these questions of power in the production of knowledge by asking what happens when community partners lead an evaluation and academics lead an evaluation of the same project. Are the research outputs produced different? And why might this be the case?
To answer these questions, the project will run workshops to supplement the evaluations. Attended by community members and academics, the first will discuss the processes, commonalities, differences, values and limitations of the various evaluation approaches chosen. The second workshop will develop a series of evaluation guidelines suited for future community-based research. Rather than being a "toolkit", this will be an exploration of the different methods used in the different contexts and what worked, or did not work, and why. This will enable future projects to make better informed decisions as to what evaluation methods they might use when planning community projects.
We cannot be prescriptive about the evaluation questions the community-led evaluation will ask. The academic evaluation will go back to original project briefs and align the original aims with the methods that the research team can use to ask whether the project has delivered what it set out to. The project also aims to explore the broader legacy of the Connected Communities programme. A key question in both evaluation sites will be the role of the AHRC as a funder of this type of activity and how this can be improved in future.

Planned Impact

The proposal is designed to deliver direct impact for the communities involved. By running two evaluations we aim to deliver a broad range of information for the groups to practically use in assessing their activities and the impact of their own activities, supporting the future development of these, especially around funding bids.
A key aim for the project is to develop evaluation guidance outlining practical examples of applied evaluation methods in use in different contexts, assessing their practicality and usefulness. The guidelines will aid projects by improving practice and helping secure future funding. We foresee a number of indirect beneficiaries of this:
Community groups involved in existing, or developing Connected Communities projects, or other academic-community collaborations: the guidelines will provide a practical tool to empower these organisations to think about evaluation themselves and discuss different methods of evaluation and measurements of success or failure with academic partners at the outset of a project.
Community groups and third sectors organisations: the dissemination of the guidelines through wider networks will support a broader range of groups. We foresee these groups being able to use the knowledge of the different methods to inform their own evaluation activities and helping them choose the right methods or tools to use from the outset.
UK Government, devolved administrations and local government: a key policy thrust across the UK is to deliver services with communities through co-production or community empowerment, either as the "Big Society" or Scotland's Community Empowerment and Renewal Bill. A more indirect benefit may be delivered through research team contacts across government who can use it to inform their own assessment of community activities, and also help them support communities to evidence their impact.
 
Title 3D Scan of Totem Pole 
Description This is a 3D scan of the totem pole erected in Wester Hailes, Edinburgh, produced using photogrammetry. 
Type Of Art Artefact (including digital) 
Year Produced 2015 
Impact The process of producing the scan brought partners in Wester Hailes. Edinburgh, (Prospect Community Housing Association) into contact with Cassiltoun Housing Association in Glasgow. There is now a developing partnership of mutual learning between the organisations. 
URL https://rms.stir.ac.uk/converis-stirling/publication/19306
 
Title Community Connections Archive 
Description This is an archive on the PARArchive website of the Community Connections Festival in Wester Hailes, Edinburgh, in June 2015. 
Type Of Art Artefact (including digital) 
Year Produced 2015 
Impact The community collaboration has continued following the activities. 
URL http://beta.pararchive.com/stories/188
 
Title The Digital Sentinel 
Description The Digital Sentinel is an online, hyper-local news source for Wester Hailes, Edinburgh. 
Type Of Art Artefact (including digital) 
Year Produced 2014 
Impact The Digital Sentinel had begun under a previous project. Under AH/L01310X/1 and AH/L01310X/2 the Sentinel was evaluated and supported to become more sustainable. It has now levered in £25,000 from the City of Edinburgh Council to continue and is offering regular IT training sessions in local community hubs. 
URL https://rms.stir.ac.uk/converis-stirling/publication/19307
 
Description This research demonstrated that co-produced research - that is research produced by academics at universities, and community partners - can have a valuable social impact. These outcomes can be measured and evaluated in a diverse range of ways from different disciplinary perspectives. Importantly for community partners, our research showed that embedding evaluation into the research process itself is very important and ensures further outcomes. Approaches such as developmental evaluation, can help build a learning culture among research partners, providing opportunities to reflect on what was working, what was not working, and providing the chance for active experiments. While embedded, community-led evaluation can provide these benefits, evaluation outputs provided by university academic partners can also have specific benefits for community partners, particularly in terms of the added legitimacy. These findings are important for how we consider the development of university-community partnerships in the future. Internationally, they will feed into the global evaluation literature and how we maximise the usefulness of evaluation to society.
Exploitation Route The 'Doing and evaluating community research - A process and outcomes approach for communities and researchers' document is publicly available through the AHRC Connected Communities website as a resource for university researchers and community research partners to use.
Sectors Government, Democracy and Justice

URL http://hdl.handle.net/1893/22080
 
Description The findings have been used by the research partners to further develop their partnership activities. This helped lead to, specifically: - levering in £25,000 of funding from the City of Edinburgh Council to sustain the Digital Sentinel news source. - levering in £35,000 per annum over three years from the Hunter Foundation to develop the Wester Hailes Community Trust from 2016. The continued activity supported by the research also sustained the impacts delivered by the prior research proposals being studied, particularly around: - embedding innovative ways of engaging with a broad range of community members by partner organisations; - continued development of partnership working among local organisations, including a successful bid for £125,000 from the Scottish Government for an innovative food and growing project in 2018 - continuing the place-making impacts of the totem pole and associated activities, specifically with a charette funded by the Scottish Government in spring 2018.
First Year Of Impact 2011
Sector Communities and Social Services/Policy
Impact Types Societal

 
Description Presentation to NCVO Third Sector Research Conference (Nottingham) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Third sector organisations
Results and Impact This was a conference presentation on the findings of the research for a practitioner and academic audience. Some follow-up with Big Local and the NCVO was attempted but was unsuccessful.
Year(s) Of Engagement Activity 2016
 
Description Scottish Third Sector Research Forum Conference Presentation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Third sector organisations
Results and Impact A conference presentation on co-producing research with communities, and evaluating co-produced researched. The audience was from across the third sector in Scotland and five national charities contacted me afterwards for further advice and support.
Year(s) Of Engagement Activity 2016