Measuring complex outcomes of environment and development interventions

Lead Research Organisation: Wildlife Conservation Society
Department Name: Grants Administration

Abstract

How do we know if development aid is spent effectively and delivering results? Aid effectiveness has become an important discussion point in international development in recent years, leading to renewed focus on how to measure the performance of development programmes, which is Theme 3 of the ESRC-DFID call. Measuring results is difficult because the results depend on what you chose to measure, when, for whom, and how it is measured. A second difficulty is attribution: how do you know that changes are due to development interventions, as opposed to other ongoing processes. Rigorous impact evaluation methodologies have been developed to address the problem of attribution by comparing intervention sites to controls. However, impact evaluation methods have tended to focus on standard economic measures of human wellbeing (e.g. income), which may not capture outcomes considered relevant by local people - such as resource tenure, education, ability to insure against shocks, or political power.

This project focuses on interventions designed to counter increasing environmental resource scarcity, caused by unsustainable use of biodiversity, deforestation, and degradation of ecosystem services, arguably some of the greatest long-term threats to human wellbeing. Understanding the processes that cause resource scarcity and what are effective interventions to address resource scarcity are relevant to Theme 1 of the call. Environment-development interventions are also some of the hardest to assess, since the processes that cause environmental resource scarcity are often very complex, and potentially affect multiple aspects of human wellbeing (not just income). For this reason, credible evaluations of their medium to long-term impacts on human wellbeing are few, providing little evidence to inform decision-makers about what works, for whom, and why.

The project aims to directly improve the implementation of policies in the environment-development sector through enhanced understanding of what works, based on more appropriate measurement of results. It brings together the Wildlife Conservation Society (WCS, a science-based environmental organization working in over 60 countries), and two of the UK's foremost academics in the field from University College London and Imperial College, as well as local and national partners. The project has three components:
(1) a methodological research component, which focuses on how to measure the development impacts of environment-development interventions on the poor
(2) a field research component, which will test the different methodological approaches with developing country researchers at 3 sites (2 in Africa and 1 in Asia) that have very different social, cultural and political settings, to see how methods perform in different circumstances and to specifically evaluate 3 field interventions focused on Payments for Ecosystem Services
(3) a practical component, which will use the results of the research to directly inform how environment-development projects are implemented in the field, building upon the consortium's networks of partners and programmes.

The research responds directly to the growing demand for knowledge about how to measure the performance of development programmes, and the specific paucity of impact evaluations in the environment-development sector. The research will therefore be high impact: both academically, in terms of publications, and for policymakers and practitioners because it responds to already-identified needs for information. The field component focuses on incentive mechanisms, such as Payments for Ecosystem Services, which have been widely promoted over the past decade as a more equitable way to ensure poverty alleviation and sustainable environmental resource management. Through the project's networks and partners, the results will directly impact how current environment-development policies are implemented, and will inform the design of future initiatives.

Planned Impact

Impact: This project addresses a central development policy issue: how can poor people's experience of change be factored more directly into policy evaluation and formulation? It delivers innovative ways to capture poor people's experienced wellbeing, enabling comparison with more established measurements of change, and building a more robust evidence base about what interventions work, how, and for whom. The project has been specifically designed with the three research sites to address issues that are a priority for local people and national governments. Through the consortium and our national partners, the project is well-placed to ensure that the results of the research are used to influence local decision-making and national policy.

Who will benefit from this research?
Besides researchers (see Academic Beneficiaries statement) other beneficiaries include: 1) national-level policymakers in the focal countries; 2) implementing organisations and practitioners; 3) local resource users; 4) the wider global network of organisations through WCS's networks and the consortium members; and 5) donor organisations and development policy initiatives.

How will they benefit from this research?
(1) National-level policymakers in Cambodia, Tanzania and Uganda will benefit from increased understanding of what works, how to measure results, and the implications of the research for existing policies (such as revenue-sharing from protected areas) and new policies (such as PES and REDD). The project sites have been chosen for their potential policy impact, looking at how insecurity of local land tenure can drive land-use change and affect poverty (Cambodia and Tanzania), how to share revenue from protected areas (Uganda), and how best to structure PES/REDD interventions to generate dual development and environment benefits (Cambodia and Tanzania).

(2) Implementing organisations and practitioners in Cambodia, Tanzania and Uganda will benefit from increased understanding of how environment and development projects are implemented and evaluated, both now and in the long-term.

(3) Local resource users will become better heard and represented, and be more willing to engage, thus improving intervention effectiveness and contributions to poverty alleviation. In all three sites, conditional, market-based, incentive payments are being used to encourage local resource users to reduce pressures on environmental resources, and increase the benefits they derive from these resources. The research will help to understand the extent to which payment interventions are achieving their stated goals, and will propose changes to improve implementation.

(4) Global networks of policy makers and practitioners will benefit from new knowledge. Project briefings, methodologies, results and publications will be distributed through consortium partners' networks with reach to >60 countries worldwide. The results will be of direct relevance to programmes in Gabon, Guatemala (funded by DFID) and Bolivia, where project partners are also measuring long-term changes in wellbeing and governance as a consequence of environment-development interventions.

(5) Donor organisations and development policy initiatives will benefit with greater understanding of what 'works' and how to measure results. The need for such information has been recognized by the High Level Forum on Aid Effectiveness, and is necessary to measure progress towards the Millennium Development Goals.

The project will build local capacity through supporting national post-graduate students with scholarships and research project supervision. All the project partners have a strong record at building national capacity, and many of these students and researchers then go on to fill high-level Government, policy and practitioner positions, ensuring lasting impact.

Publications

10 25 50

publication icon
Brockington D (2015) Protected areas and poverty. in Philosophical transactions of the Royal Society of London. Series B, Biological sciences

publication icon
Clements T (2015) Impact of payments for environmental services and protected areas on local livelihoods and forest conservation in northern Cambodia. in Conservation biology : the journal of the Society for Conservation Biology

publication icon
Clements T (2014) Impacts of Protected Areas on Local Livelihoods in Cambodia in World Development

publication icon
Milner-Gulland EJ (2014) Accounting for the impact of conservation on human well-being. in Conservation biology : the journal of the Society for Conservation Biology

publication icon
Woodhouse E (2015) Guiding principles for evaluating the impacts of conservation interventions on human well-being. in Philosophical transactions of the Royal Society of London. Series B, Biological sciences

publication icon
Wright JH (2016) Reframing the concept of alternative livelihoods. in Conservation biology : the journal of the Society for Conservation Biology

 
Description The key finding of our ESRC supported research is that we determined that there are low cost but credible ways to both track trends in multiple dimensions of human wellbeing over time, and attribute these positive or negative changes to a conservation or development project's activities.

Our research showed that it is relatively easy to combine a locally relevant measure of the multiple dimensions of human wellbeing with interviews where project recipients talk about the wellbeing changes that have occurred in their lives and in their communities. This blending of the quantitative Basic Necessities Survey with qualitative ethnographic information offers a cost effective approach for both documenting and reporting wellbeing trends, and simultaneously assessing the influence conservation and development actions had on observed wellbeing trends.

These key findings are a significant advance in our ability to evaluate the impact of conservation and development project activities on human wellbeing.
Exploitation Route Our findings were translated into a relatively short user-friendly manual that explains clearly how conservation and development practitioners might both track trends in multiple dimensions of human wellbeing and simultaneously assessing the influence conservation and development actions had on observed wellbeing trends.
Sectors Agriculture, Food and Drink,Communities and Social Services/Policy,Environment

 
Description USAID CARPE adopted the project's advocated approach to assessing human wellbeing trends over time (the Basic Necessities Survey). https://programs.wcs.org/DesktopModules/Bring2mind/DMX/Download.aspx?EntryId=28643&PortalId=97&DownloadMethod=attachment
First Year Of Impact 2015
Sector Environment
Impact Types Policy & public services

 
Description ESPA - this grant was awarded to Dr. Emily Woodhouse
Amount £0 (GBP)
Funding ID NE/P008097/1 
Organisation Natural Environment Research Council 
Sector Public
Country United Kingdom
Start 12/2016 
End 11/2017
 
Description Imperial College ESRC Impact Acceleration Account
Amount £12,000 (GBP)
Organisation Economic and Social Research Council 
Sector Public
Country United Kingdom
Start 01/2015 
End 12/2015
 
Title Guide to the modified Basic Necessities Survey 
Description A locally relevant tool for assessing human wellbeing 
Type Of Material Physiological assessment or outcome measure 
Year Produced 2015 
Provided To Others? Yes  
Impact Adopted by USAID as the human wellbeing assessment method for their Central African Regional Program for the Enviroment 
URL http://globalinitiatives.wcs.org/DesktopModules/Bring2mind/DMX/Download.aspx?EntryId=28643&PortalId=...
 
Description Collaboration with IIED 
Organisation International Institute for Environment and Development (IIED)
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution PI Dr. David Wilkie is working in collaboration with Dr. Dilys Roe, International Institute for Environment and Development (IIED), for their Defra, UK UK Darwin Initiative Award Ref:20010 Social Assessment of Protected Areas (SAPA), to jointly implement the research program in the Republic of Gabon with WCS and the Agence Nationale des parcs nationaux du Gabon. More generally we are working with IIED to share knowledge, methods and research outputs.
Collaborator Contribution IIED and WCS will work jointly on producing synergistic research outputs relevant to both research projects, which have complementing aims.
Impact Field research is underway, and meetings between the SAPA and our project teams have been held to find common ground, and share understandings. The collaboration is multi-disciplinary in that both teams consist of social and natural scientists.
Start Year 2014
 
Description Collaboration with Sustainable Use and Livelihoods Specialist Group - IUCN 
Organisation IUCN, International Union for Conservation of Nature
Country Switzerland 
Sector Charity/Non Profit 
PI Contribution Help IUCN-SuLi develop a conceptual model for human wellbeing impacts of anti-poaching and anti-trafficking investmetns
Collaborator Contribution Help IUCN-SuLi develop a conceptual model for human wellbeing impacts of anti-poaching and anti-trafficking investmetns
Impact None specific
Start Year 2013
 
Description PRISM Project 
Organisation University of Cambridge
Department Cambridge Conservation Initiative
Country United Kingdom 
Sector Academic/University 
PI Contribution Started a collaboration with the PRISM project: Practical Impact Assessment Methods for Small and Medium-sized Conservation Projects, funded by Cambridge Conservation Initiative. Will be feeding some of the learning from our ESRC-DFID project into their products.
Collaborator Contribution Guidelines for assessing human wellbeing impacts of conservation projects
Impact None
Start Year 2014
 
Description Poverty and ecosystem Impacts of payment for wildlife conservation initiatives in Africa: Tanzania's wildlife Management Areas (PIMA) 
Organisation University College London
Country United Kingdom 
Sector Academic/University 
PI Contribution Assisted in the development of a human wellbeing assessment tool
Collaborator Contribution None
Impact Several peer reviewed papers
Start Year 2013
 
Description Basic Necessity Survey (BNS) Introduction and Planning Workshop for WCS Regional Drivers Sites 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact We held a workshop for 10 project managers/technical advisors from WCS in South East Asia (Myanmar, Laos and Cambodia programmes) to build capacity and understanding in research methods for social impact evaluation.





Working towards adoption of the Basic Necessities Survey methodology by other country programmes in Asia (beyond the case study in Cambodia), as best practice across the regional landscape.
Year(s) Of Engagement Activity 2014
 
Description Blog on Imperial College Conservation Science (ICCS) website 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact We posted a blog introducing the ideas upon which the project is based on a well read website which was shared through social media (facebook, twitter) and through ICCS's network. It is read by conservation / natural resource students, researchers and practitioners.

None yet.
Year(s) Of Engagement Activity 2013
URL http://www.iccs.org.uk/money-money-money/
 
Description Blog on conservationmagazine.org and National Geographic 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact The blog which draws upon the theory behind the research project on robust impact evaluation methods was posted on two widely read science website and shared on social media.

No specific impacts known.
Year(s) Of Engagement Activity 2014
URL http://voices.nationalgeographic.com/2014/03/07/why-adoption-of-the-medical-model-would-cure-conserv...
 
Description Defining human wellbeing for conservation projects in northern Cambodia 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Over 50 people attended the presentation at the International Congress for Conservation Biology , August 2015
Year(s) Of Engagement Activity 2015
 
Description Lecture at Clark University Department of Geography 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact The talk which introduced our methods and approach to graduate students resulted in interest and discussion.

No specific impacts so far.
Year(s) Of Engagement Activity 2014
 
Description Lecture at Imperial College London 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Postgraduate students
Results and Impact ~ 40 post-graduate students studying for MSc in Conservation Science attended a session 'Human wellbeing & conservation: evaluating the social impacts of interventions' which included a presentation and interactive group activities on understanding and measuring the impacts of conservation on human wellbeing based on the ESRC-DFID funded research. It stimulated lively discussion and thoughts amongst the group who are training to be conservation researchers and practitioners.

The lecture stimulated interest in research projects which students carry out in the summer term, and led to the completion of a project which has built skills and knowledge for the student and will contribute to the research outcomes of the ESRC-DFID funded project.
Year(s) Of Engagement Activity 2013,2014
 
Description Lecture at Yale University School of Forestry 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact The lecture 'Measuring human wellbeing impacts of conservation and development projects' was followed by a Q&A session and discussion.

None so far.
Year(s) Of Engagement Activity 2014
 
Description Lecture at the Royal University of Phnom Penh 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Approximately 30 Cambodian post-graduate students studying for an MSc in Biodiversity Conservation attended the talk which introduced them to ideas and details of our research on understanding social/wellbeing impacts of conservation interventions. It sparked discussion and questions from the students who have little experience of social science perspective and are likely to become leaders in the field of conservation and development in Cambodia.

None yet.
Year(s) Of Engagement Activity 2013
 
Description Lecture to Royal University of Agriculture, Cambodia 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Undergraduate students
Results and Impact The lecture was to approximately 200 3rd and 4th year undergraduate students, and introduced them to the research and wider concepts of social impact evaluation, leading to discussion and questions.


The lecture instigated the development of a working relationship with the Ecoland team at RUA http://ecolandcambodia.blogspot.fr/p/about-ecoland.html. We are in discussions about a potential collaboration between RUA and WCS to provide support to undergraduate student projects on the topic of conservation and well-being.
Year(s) Of Engagement Activity 2014
 
Description Methods for evaluating the human wellbeing impacts of nature conservation interventions 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Two day workshop on "Methods for evaluating the human wellbeing impacts of nature conservation interventions" 16th & 17th June at University College London.
Year(s) Of Engagement Activity 2015
 
Description Op-ed on livescience 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Public/other audiences
Results and Impact The post entitled 'Conservation Is About Caring for Nature and People' was an "Expert Perspective Op-Ed' and drew upon the perspective emerging from the research project. It was shared widely on facebook and twitter.

No specific impacts yet.
Year(s) Of Engagement Activity 2013
URL http://www.livescience.com/40927-conservation-is-about-caring-for-nature-and-people-op-ed.html
 
Description Presentation at the World Parks Congress 2014 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact The session on measuring the impacts of conservation was well attended by ~150 people and was followed by questions and answer session.

GBIF (Global Biodiversity Information Facility) and GEF (Global Environment Facility) Evaluation office requested more details and a follow up discussion of the applicability of the proposed 5 measures approach to their programs.
Year(s) Of Engagement Activity 2014
 
Description Presentation titled "Evaluating the impacts of conservation on human well-being" ESRC-DFID Conference Pretoria, South Africa, March 2016 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Evaluating the impacts of conservation on human well-being
Year(s) Of Engagement Activity 2016
 
Description Presentation to USAID in Washington DC 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact The presentation led to further discussions and impacts (see below).


The presentation led to the policy decision by USAID require partners (WCS, WWF, AWF, and CI) to adopt the approach of using five measures (including governance and wellbeing) to monitor progress of the Central African Regional Program for the Environment (CARPE) during phase III.
Year(s) Of Engagement Activity 2013
 
Description Seminar at University of Colorado 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Postgraduate students
Results and Impact Project lead on the Tanzania case study Prof. Terrence McCabe held a seminar with anthropology graduate students (Environment and Society Program) introducing them to the case study aims, methods, and understandings so far. This generated discussion and interest in the project and broader issues concerning conservation and development.

As a result of the seminar, one graduate student is planning on conducting his dissertation on the research ideas of conservation and wellbeing.
Year(s) Of Engagement Activity 2014
 
Description Seminars at UK universities (Cambridge, Oxford, York, Aberdeen) on 'How do we know if we're making a difference: Finding solid ground in the shifting sands of conservation evaluation 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact The seminar sparked interest and discussion about the topic.

An NGO practitioner subsequently contacted the project co-PI in response to the seminar, and has received support on a grant proposal for a conservation intervention related to the topic of the seminar (specifically use of mobile technology for improved monitoring and evaluation of natural resource projects) . A student contacted the project co-PI, and has developed her PhD proposal in response to ideas in the seminar.
Year(s) Of Engagement Activity 2014
 
Description Talk at LRMC Biodiversity and Health symposium, Phnom Penh 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact The overall aim of the symposium was share the scientific and technical expertise linking biodiversity and health concerns. The talk provided an opportunity to share and discuss our research methods with practitioners and academics in linked fields of study of health and development, develop our relationship with development donors and practitioners in Cambodia.



We have discussed a potential follow-up publications on health, ecosystems and human wellbeing as a result of the roundtable discussion.
Year(s) Of Engagement Activity 2014
URL http://christopheboete.net/science/biodivhealthPNH/index.html
 
Description Talk at the Society for Conservation Biology Asia conference 2014 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Type Of Presentation paper presentation
Geographic Reach Regional
Primary Audience Professional Practitioners
Results and Impact The talk entitled 'Evaluating impacts of conservation projects: why does wellbeing matter? focused on the Cambodia case study. It resulted in follow up discussions with practitioners from other NGOs to learn more about our methodology and by academics to exchange ideas.

Nothing specific so far.
Year(s) Of Engagement Activity 2014
URL http://scbasia2014.org/?page_id=518
 
Description What is good evidence? 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Participation in Darwin Initiative Evidence Workshop: What is 'good evidence'? International Institute for Environment & Development, October, 2015.
Year(s) Of Engagement Activity 2015
 
Description What is human wellbeing? A three-dimensional perspective 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Presentation at an expert workshop on Equity, Justice and inclusion in ecosystem governance. International Institute for Environment & Development, London, March 2015. Presentation: 'What is human wellbeing? A three-dimensional perspective'
Year(s) Of Engagement Activity 2015