The diffusion of development: Extending RCTs with agent-based modelling to understand spillovers of development interventions
Lead Research Organisation:
Royal Holloway University of London
Department Name: Politics, Internatl Relations & Philos
Abstract
Randomised controlled trials (RCTs) are widely considered the gold-standard to assess the impact of development interventions. However, RCTs can only go so far in helping development organisations answer the key question that interests them: Do their interventions improve people's livelihoods? After all, experimental impact evaluations can only tell us how treated individuals fare in comparison to control individuals. Meanwhile, development organisations implement interventions in the hope that benefits extend beyond their direct beneficiaries. Because it is impossible to provide aid, training, or other assistance to everyone who needs it, such organisations often hope for spillover effects of their interventions that have long-lasting and self-sustainable effects on the whole community despite only targeting a few individuals. Moreover, it is plausible that some interventions require spillovers as individuals need support from their communities to make life-changing decisions.
We propose to transform the ways that researchers and development organisations assess the impact of development interventions by combining RCTs with agent-based computational modelling (ABM). We hope to apply agent-based modelling in a novel way, by using them to enhance RCTs and providing a cost-effective measure of the widespread effects of interventions beyond the direct beneficiaries.
This methodology greatly improves upon existing experimental designs that are adapted to study spillovers - not only because it is more flexible and more cost-effective, but because it can reveal much more about spillover effects. First, existing experimental designs only allow us to observe spillovers on strictly defined units. Our method will allow us to observe population-wide spillovers. Moreover, because we can observe population-wide spillovers, we can see how changes to interventions can shape outcomes at the macro-level. And finally, while we know that the interventions will often work or work best when a sufficient number of individuals have adopted the desired behaviour, it is very difficult to observe where these thresholds lie using existing experimental designs. Our method will allow us to observe tipping points at which the treatment has affected sufficient numbers of people that subsequent change is readily accepted.
We will apply our proposed methodology to a development intervention - funded by DfID and implemented by the International Organisation for Migration (IOM) -- aimed at improving prospects for self-employment in the Gambia and Senegal. Our ultimate aim is to provide transparent and thorough guidelines on appropriate experimental designs that can capture community effects; programming packages to implement the methodology in NetLogo, a free and easy-to-use ABM platform; and a user-friendly app to conduct preliminary tests and power calculations.
We propose to transform the ways that researchers and development organisations assess the impact of development interventions by combining RCTs with agent-based computational modelling (ABM). We hope to apply agent-based modelling in a novel way, by using them to enhance RCTs and providing a cost-effective measure of the widespread effects of interventions beyond the direct beneficiaries.
This methodology greatly improves upon existing experimental designs that are adapted to study spillovers - not only because it is more flexible and more cost-effective, but because it can reveal much more about spillover effects. First, existing experimental designs only allow us to observe spillovers on strictly defined units. Our method will allow us to observe population-wide spillovers. Moreover, because we can observe population-wide spillovers, we can see how changes to interventions can shape outcomes at the macro-level. And finally, while we know that the interventions will often work or work best when a sufficient number of individuals have adopted the desired behaviour, it is very difficult to observe where these thresholds lie using existing experimental designs. Our method will allow us to observe tipping points at which the treatment has affected sufficient numbers of people that subsequent change is readily accepted.
We will apply our proposed methodology to a development intervention - funded by DfID and implemented by the International Organisation for Migration (IOM) -- aimed at improving prospects for self-employment in the Gambia and Senegal. Our ultimate aim is to provide transparent and thorough guidelines on appropriate experimental designs that can capture community effects; programming packages to implement the methodology in NetLogo, a free and easy-to-use ABM platform; and a user-friendly app to conduct preliminary tests and power calculations.
Publications
| Description | Our findings from the research are based on three distinct arms of the project: an empirical analysis of the impact evaluation, focus groups with practitioners, and the design and programming of the agent-based model. With regards to the first goal, we conducted a rigorous experimental evaluation of the IOM's youth entrepreneurial program to test the programme's effects on migration aspirations. We find that the program did reduce migration aspirations, but primarily in the short-term. Through multiple mediation analysis, we conclude that the primary acting mechanism through which aid shifts migration attitudes is place attachment (or the perception that my place of residence is better than anywhere else to achieve my goals), rather than self-efficacy or self-sufficiency. To the extent that individuals grow in self-confidence without forming a deep attachment to the local community, such interventions will struggle to shape migration decisions. Beyond these programme-specific results, the designed a tool for examining the spillovers of any development intervention. To establish a foundation for the model, we conducted two focus groups with key stakeholders. The first group consisted of individuals who propose, design, and implement development interventions, and the second consisted of professionals involved in the monitoring and evaluation aspect of such interventions. We concluded that many development practitioners refer to spillovers in their work, but often under different names; there is no consistent terminology or understanding throughout the sector. We found that spillovers are frequently discussed at the planning stages of an intervention, particularly in terms of risk assessments. Also, many funders expect a consideration of spillovers to be embedded into their proposals. However, empirical evaluations of spillovers are practically non-existent, and most practitioners argue that such an evaluation would be impossible. They are so tightly constrained in terms of the data they can collect that they balk at the notion of collecting data from undefined non-beneficiaries. It is with this input from practitioners that we programmed an agent-based model to evaluate intervention spillovers. The model has three aims: 1) to identify the effects of development programmes beyond programme beneficiaries, 2) to do so in the context of a traditional RCT-based evaluation, using only the data that development practitioners would realistically be expected to collect, and 3) to provide guidelines that would help practictioners and evaluators to think systematically about spillovers as effects of their programmes. In regards to the first aim, our model produces important key outputs, such as the number of non-beneficiaries "treated" and the differences in outcomes between treatment and control contacts. Our model includes a setting which is ambivalent to geography, in addition to a setting which varies the probability of contact over distance. For the second aim, our agent-based model represents a tool that could be used to examine possible spillovers for any outcome where appropriate date are collected. To achieve that, we compile a concise list of survey items that would be required to calibrate the model at baseline and endline data collection stages. The empirical analyses of these survey items can reveal interesting trends in their own right: such as descriptive data on the relevant networks of programme beneficiaries, whether treatment or control respondents are more likely to initiate spillovers, and whether people learn more from successes or failures of their network contacts. For the third aim, we define two different types of programme spillovers - indirect effects and observational spillovers - and provide clear guidance with examples to help practitioners anticipate them and collect data on them. |
| Exploitation Route | This model was designed with input from development practitioners, and even at this stage in the research, we believe they could put our work to use. One way that others might use the outcomes of this project is by incorporating our spillover questionnaire into impact evaluations. Our focus groups highlighted how difficult it is to even think about these sorts of effects, and we demonstrate that a few key survey questions can go a long way in a measuring spillovers. Regarding the model itself, we believe practitioners could benefit from the research by developing systematic and clear expectations about spillovers and the mechanisms by which they could take place. Our focus groups indicate that such systematic conceptualization would be especially helpful at the proposal stage of an intervention and when Monitoring and Evaluation is being planned. We believe the best way to take this funding forward would be to apply it in additional contexts with other interventions. The mechanisms through which spillovers are expected to take place can vary by intervention, and it will be important to consider these mechanisms as extensions to the model. We are currently discussing opportunities with practitioners. |
| Sectors | Government Democracy and Justice |
| Description | We have developed a Community of Practice on Migration Spillovers. The aims of the CoP are two-fold: i) further develop and improve our methodology for measuring spillovers, and ii) to springboard its dissemination and impact. To organize and maintain the CoP, we are partnering with the London office of the International Organization for Migration (London Office). The IOM has shown great enthusiasm with our project, and the Chief of Mission in London has highlighted our research with the Home Office. Additional members come from the FCDO, the Laudes Foundation, Open Society, the Overseas Development Institute, and British Council, and RapiData. The CoP is expected to hold three in-person sessions from 2023-2024. Two arleady took place in June 2023 and March 2024, and a third is planned for September 2024. The output of our first session was an SOP (Standard Operating Procedure policy document) and the output of our second was a spillover survey module that could be used by M&E specialists in their impact evaluations. |
| First Year Of Impact | 2023 |
| Sector | Government, Democracy and Justice |
| Impact Types | Policy & public services |
| Title | Spillover ABM: An data-driven agent-based model to extend RCT analysis to spillovers |
| Description | This model seeks to extend what researchers can learn from an RCT-based impact evaluation by modelling spillover effects. It uses baseline and endline survey data, drawn from programme participants and a control group. The basic model considers intended spillovers, downstream effects of an intervention that are transmitted deliberately and explicitly by treated subjects onto the non-target population. Quantities of interest resulting from the basic model include the number of people who are indirectly impacted by beneficiaries, the length of the spillover chain, and the difference in outcomes between those who were impacted directly, impacted indirectly, and not impacted at all. The model can be set as one large geographic cluster, or it can account for multiple clusters that allow researchers to examine geographic distance of the spillovers. The next planned stage of the model will examine unintended spillovers, or the effects that take place as the non-target population observes the positive or negative outcomes of the treatment. |
| Type Of Material | Computer model/algorithm |
| Year Produced | 2022 |
| Provided To Others? | No |
| Impact | At this early stage, we have not observed noticeable impact. However, we are discussing opportunities to use our model in subsequent impact evaluations and seek to validate it under different conditions. |
| Description | Spillover knowledge product |
| Form Of Engagement Activity | Engagement focused website, blog or social media channel |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Third sector organisations |
| Results and Impact | We worked with a professional artist to compile a knowledge product, widely available to all workshop participants and easy to disseminate, that summarized the research. It described and the motivation behind the ABM tool, the relevant focus group discussions, and the architecture in a highly digestible, self-contained form. |
| Year(s) Of Engagement Activity | 2022,2023 |
| Description | Spillover workshop with practitioners |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Third sector organisations |
| Results and Impact | We conducted a workshop for an audience of roughly 40 practitioners, in which we described the results of our impact evaluation and introduced our agent-based modelling tool. Participants came from a variety of organizations, include the Home Office, the FCDO, the IOM, the Laudes Foundation, among others. The workshop was divided into three sections: 1) an overview of the impact evaluation, 2) an analysis of the focus groups we conducted to co-design the ABM tool, and 3) an introduction to our ABM tool - its architecture, interface, and the results as they apply to our impact evaluation. The workshop resulted in a number of queries for additional work, and a final discussion led us to believe that participants were enthusiastic about the model and its potential. |
| Year(s) Of Engagement Activity | 2022 |