Monitoring and Evaluation for Sustainable Communities

Lead Research Organisation: University of Oxford
Department Name: Geography - SoGE

Abstract

Community groups do have a pivotal role to play in meeting the UK's ambitious greenhouse gas emission reduction commitments and sustainable development goals. Across the UK there are currently over 600 'low carbon' community groups engaged in activities that aim to reduce emissions, and bring about broader sustainability changes at the local level. However, there are currently issues with acquiring rigorous data about the range of impacts and outcomes achieved by these groups. That is, while community groups are without doubt having positive effects 'on the ground', evidence from recent research suggests that more robust monitoring and evaluation (M&E) of processes and outcome through co-developed resources would enable groups to identify their impacts, and clarify priority areas for future action. In addition, while a range of M&E tools and resources do exist, these resources often fail to be applicable to the needs, contexts, and capacities of community groups. Therefore, if groups were able to implement comparable, clear and relevant M&E, a more complete picture of the contributions they collectively make to national targets could emerge, which in turn would feed into broader debates about UK sustainability and climate change mitigation, adaptation and resilience.

In response, this project is a collaboration between social scientists at the University of Oxford, and two partner organisations that coordinate low carbon community action in the UK: Transition Network (TN) and Low Carbon Communities Network (LCCN). The project aims to bring the latest social science research about M&E into dialogue with the needs and current practices of these Networks and related groups. Specifically, this project will

- collaborate with TN, LCCN, and a sample of affiliated community groups to identify their M&E priorities and needs;
-develop appropriate M&E resources in response;
-build capacity within local groups to pilot and implement these resources;
-collect data on process, experiences and outcomes from the above;
-produce transferable material that can be used and adapted by other groups and organisations with similar aims.

To achieve these aims, there is a four-stage activity programme planned, as follows:

Stage 1: Two workshops will present and discuss current M&E resources with Networks and related groups, to gather information on experiences and needs, and recruit groups to participate in later Stages. From this information, and in conjunction with project partners, appropriate M&E resources will be developed.

Stage 2: Two workshops will build capacity amongst groups to use M&E resources. A complementary online platform will allow participation from a broader range of groups beyond the workshops.

Stage 3: The piloting and implementation of above M&E resources amongst participating groups, with researchers offering support and collecting data on experiences and outcomes. Resources will also be shared online through the development of a web-based platform. The aim is for 10 -15 groups to pilot a range of M&E resources intensively with the full involvement of their local groups: whilst a further 20 groups will pilot selected M&E resources according to their needs.

Stage 4: Two final workshops to bring groups together to provide feedback on process, experiences, findings, and next steps.

These activities aim to maximise the exchange and development of M&E knowledge, resources and experience, to ensure relevant and user-orientated outputs for academics, project partner organisations, involved communities, and other users/beneficiaries.

Planned Impact

Communities have a pivotal role to play in meeting the UK's ambitious greenhouse gas emission reduction commitments and sustainable development goals. To do so it is vital that groups are able to monitor and evaluate the outcomes of their actions in a grounded and self-sustaining way, which will enable them - along with coordinating networks - to make meaningful and valid contributions to national-level debates. As such, this project has a wide range of beneficiaries, who are:

1. Academics and researchers, particularly those interested in action learning research with communities; monitoring and evaluation (M&E) of low carbon/community groups; and best practice processes for community self-evaluation.
2. Coordinating and support bodies, in particular project partners Transition Network (TN) and Low Carbon Community Network (LCCN).
3. Participating community groups, through ongoing engagement and the piloting of relevant resources within their local groups.
4. Other community groups, including those not affiliated with TN and LCCN who will be able to access M&E resources online.
5. Policy makers and Think Tanks interested in the community-level mitigation and adaptation.
6. Media: particularly those interested in low carbon and community-level sustainability, and how this feeds into national policy and mitigation/adaptation strategies.
7. Members of the public with an interest in community-level change, M&E and/or UK mitigation and resilience policy.

How will they benefit?
1. Access to, as well as synthesis and translation of, published and ongoing research into low carbon community initiatives and M&E resources.
2. The creation of participatory workshop space and processes to articulate and build relevant and user-friendly M&E resources.
3. Capacity building for M&E amongst participating groups.
4. Access to grounded and piloted M&E resources for both participating groups and other interested groups via online availability.
5. Dedicated online spaces to learn about, and reflect upon, community-level M&E.
6. Opportunity for TN and LCCN to embed rigorous M&E into ongoing organisational practices.
7. Availability of piloted aggregate data on collective community impacts, to provide evidence base for policy making.
8. User-focused reports and academic papers/conference presentations.
9. Strengthening of existing, and establishment of new, working relationships amongst academics, project partners, advisory network members and participating groups.

What are the impacts?
This project contributes directly to the ESRC's Strategic Priorities, and thus UK economic and social life. In particular, it contributes to the theme of 'Influencing Behaviour and Informing Interventions' by helping to improve community-level sustainability and the work of participating coordination Networks: and 'A Vibrant and Fair Society', by enhancing the contribution the voluntary sector makes to low carbon development. The impacts will include:
1. The wide dissemination of academically-informed, co-developed and piloted M&E resources.
2. Better informed public debate about, and policy making concerning, the role of communities in low carbon futures in the UK.
3. TN and LCCN groups that are able to better understand their impacts, and thus create programmes and projects to maximize future impacts.
4. Establishment of strategic knowledge exchange partnerships, to be maintained through ongoing working relationships and/or future research.
5. Contributions to academic debates through empirically-grounded research that includes the input from communities and coordinating networks.

Publications

10 25 50
 
Title Project website 
Description Website constructed for use by project participants, to download M&E tools to be trialled 
Type Of Art Artefact (including digital) 
Year Produced 2014 
Impact Participants have been able to access a wide range of downloaded M&E tools 
URL https://mescproject.wordpress.com/
 
Description This project sough to examine and trial the feasibility of UK low carbon community groups collecting their own monitoring and evaluation data. Main findings are:
1. Seemingly simple monitoring and evaluation tools can have a significant positive impact on the groups using them, in terms of these groups being able to plan better; to consider their own progress and sustainability; and see what aspects of their external impact needs further work, in reference to their aims and goals.
2. These tools however need to be adaptable and flexible, to mirror the needs and skills within the group. As there is a such a variety of low carbon community groups-in terms of size, focus, composition, and longevity-attempting to create 'one size fits all' monitoring and evaluation tools is neither feasible nor desirable.
3. Such flexibility and adaptability diminishes the potential to collect comparable aggregate data across similar groups or a movement. However, the trailing of a template questionnaire for groups to send out to their members did generate some comparable data, particularly on the issues of participants involvement in/perception of the group, motivations for involvement; actions taken or intended to be taken through involvement in the group.
4. Although creating online tools is useful, groups need training and support in creating and undertaking their own monitoring and evaluation. This includes having a 'champion' within their group who is able to undertake the additional tasks that monitoring and evaluation requires, as it can be time-consuming and attract additional costs to the groups running.
5. There are still open debates about the best way to evaluate the social impacts of groups' activities, particularly around issues of the distribution of social and economic benefits.

Materials and processes developed from this project:
1. Material for, and the process of, a 1-day workshop to help low carbon community groups formulate their own monitoring and evaluation strategies. These workshops were run 5 times and have been iteratively improved over that time.
2. 12 monitoring and evaluation tools for low carbon community groups, which enable them to gather data on internal group processes and structure, and external impact. These tools have been trialled over several months by 20 groups around the UK, to evaluate their usefulness and the quality of data collected. Feedback from trailing groups has been used to improve these tools, which are currently located on the project website (see below) for anyone to access and download.
3. Whilst most of the above tools were developed from existing resources, two new tools were created by this project, to measure Equity impacts, and Cost Effectiveness of low carbon community groups' projects. These tools were trialled by one Bristol-based network, and have been improved from this feedback, though further work on their development is required.
Exploitation Route The findings and outputs have been taken forward by the Transition Network: a central partner in this project. Through 2016, the PI worked with Transition to incorporate this projects' findings and tools into their on-going redevelopment of the support they offer Transition groups, as well as Transition's detailed web presence. This new package was launched by Transition earlier this year (2017) and they have reported to us it is proving successful. In addition, the monitoring and evaluation tools developed during the project trials could be developed even further. In particular, the Equity Impact and Cost Effectiveness tools produced some interesting data when trialled, but requires more field testing and methodological development before they could be widely utilised. Indeed, the PI is part of an ongoing conversation with the organisation Leapfrog, about the possibility of undertaking this work in the near future. Finally, finding ways to build vital capacity for monitoring and evaluation within low carbon community groups presents a challenge and requires some creative thinking, such as possible peer-to-peer training and processes or regional specialists.
Sectors Communities and Social Services/Policy,Digital/Communication/Information Technologies (including Software),Education,Environment,Government, Democracy and Justice

URL https://mescproject.wordpress.com/
 
Description Our findings have been used in the following ways: 1. Project participants: participation in this project has had significant positive impacts on low carbon community groups trailing the monitoring and evaluation tools. They have reported to us, in extensive feedback taken towards the end of the project, the value of the capacity-building and training undertaken, both for the sustainability and well-being of their group and for the impact they are having on the wider community. It has also started conversations between groups and among regional networks, about the importance of Monitoring and Evaluation to future actions, and various regional hubs (such as the Bristol Energy Network) are putting in place regular and semi-formalised Monitoring and Evaluation processes 2. Networks: this project partnered closely with the Transition Network and, less so, with the Low Carbon Communities Network. The Transition Network has reported the notable impact this project has had on the time, skills and capacity they have to be able to help their thousands of groups around the world monitor and evaluate themselves. Aspects of this project have been taken up by Transition Network, as a key part of its training and support to groups. In addition, in early 2017 Transition took over hosting of some of the tools from this project, to further embed them in their organisation. This means that the tools package developed by this project is being put into direct use and implementation by one of the key global networks for community environmental groups. In February 2017, the PI received a follow-up email from Transition, stating how useful the tools are proving for them, to help monitor their groups and the growth of their network. 3. Further impact: a comprehensive project report was completed and circulated to project participants, including members of the Advisory Network who are influential practitioners in this field. One full length paper has been published Environment and Planning A. Another paper is under preparation. Finally, the PI has been invited to various events to report on project and connect with others doing similar work i.e. the Towards European Societal Sustainability conference in Edinburgh in May 2015: a workshop help by Nesta into Monitoring and Evaluation in September 2015; and 1-day seminar on community research at Luxembourg University in December 2016. In addition, she was approached in 2016 by the community energy group Leapfrog, and presented a seminar on 27th September 2016 in London, about the social impacts of community energy. Up to March 2017, there are on-going conversations about the PI working with Leapfrog on this issue, in the near future.
First Year Of Impact 2015
Sector Communities and Social Services/Policy,Environment
Impact Types Societal

 
Description Community Energy Strategy
Geographic Reach National 
Policy Influence Type Citation in other policy documents
URL https://www.gov.uk/government/publications/community-energy-strategy
 
Description Impact on Pure Leapfrog
Geographic Reach National 
Policy Influence Type Influenced training of practitioners or researchers
Impact The PI was invited, in September 2016, to present the findings from this project to Pure Leapfrog (PL): 'providers of social investment to community energy projects in the UK (see https://www.pureleapfrog.org/Pages/Category/project-funding). At that time, PL was in the process of creating a Monitoring and Evaluation framework for their funded projects. After this presentation, the PI maintained contact with PL, giving them written and oral feedback on their M&E framework, drawing on experience from this project. This feedback was instrumental to PL creating a robust framework, which is now being utilised with funded communities around the UK: helping these communities understand and track their impact, and giving much -needed data on effectiveness to PL and their funders.
 
Description Transition Training
Geographic Reach National 
Policy Influence Type Influenced training of practitioners or researchers
 
Title Results of feedback from the monitoring and evalation trials from groups 
Description This data takes the form of qualitative feedback from groups who had trailed the monitoring and evaluation tools. There are 16 interview transcripts: as well as responses to an online questionnaire that asked for detailed feedback on the specific tools. The raw data of these questionnaires are available, as is an Excel spreadsheet that collates these responses. 
Type Of Material Database/Collection of data 
Provided To Others? No  
Impact It is the first time that data has been collected directly from low carbon community groups, about their experiences undertakening their own monitoring and evaluation on their group, and their groups impact on the wider community. 
 
Description Advisory Panel 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? Yes
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Generated resource sharing, lively discussion and feedback on early project outputs

Have fostered working relationships with experts in the field and generated further interest in the project, through their professional networks and activities
Year(s) Of Engagement Activity 2014
 
Description Invited paper Luxembourg University 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact About 50 people attended an open 1-day workshop on researching communities, held at the University of Luxembourg (December, 2016). I presented an invited paper, which created a lively debate: and I have subsequently been invited into a small, international working group, to put together a future funding bid around this theme.
Year(s) Of Engagement Activity 2016
 
Description Keynote at Institute of Australian Geographers conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact After this keynote, there were several questions from the audience plus conversations afterwards which have put me in contact with interested researchers in Australia and New Zealand

I have had follow-up from several people, including requests for further infomration about research activities and outputs.
Year(s) Of Engagement Activity 2015
URL https://kaigi.eventsair.com/QuickEventWebsitePortal/iag2015/iag2015/ExtraContent/ContentPage5
 
Description Low Carbon Community Groups 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Public/other audiences
Results and Impact Lively engagement with participants and uptake of monitoring and evaluation tools

Facilitated low carbon community groups present to rethink their projects and strategies, including how to create projects that can be easily monitored and/or evaluated. Feedback stated it has helped their group develop and sustain itself. Also recruited some groups to take part in a 3 month trial of M&E tools.
Year(s) Of Engagement Activity 2014
 
Description SCORAI talk 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact Generated many questions from workshop participants about project.

As a result of this talk, the PI Kersty Hobson was invited and attended a subsequent workshop in Switzerland, to develop the research ideas further
Year(s) Of Engagement Activity 2013
URL http://scorai.org/cfp-now-open-istanbul-2013/
 
Description Talk at Nesta 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact This was an invited talk at an afternoon workshop on Monitoring and Evaluation, organised and held at Nesta in London. There were questions and interest in this project, from audience members, and due to this Kersty Hobson is now in communication with researchers and practitioners around the UK interested in undertaking similar work, to share learning and outputs.
Year(s) Of Engagement Activity 2015
 
Description Talk to Leapfrog. London 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact I was invited by the organisation Leapfrog to present a talk on monitoring and evaluation of community energy, in London (September, 2016). The audience was a mixture of third sector funders, community group leaders and researchers. Since then, I have had on-going dialogue with Leapfrog, who are now taking this agenda forward within their own organisation, with the aim to undertake further work in researching the social impacts of community energy in collaboration.
Year(s) Of Engagement Activity 2016
 
Description Talk to TESS conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact This talk provoked several questions from the floor about the methods and also outcomes.

Impacts include plans for future collaborations with individuals met at this conference, and requests for further information about project activities and outputs
Year(s) Of Engagement Activity 2015
URL http://www.tess-transition.eu/conference/