How different are direct mail and telephoning? A 'Get Out The Vote' randomised experiment in the 2009 European and Local elections

Lead Research Organisation: University of Manchester
Department Name: Social Sciences

Abstract

Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
 
Description We provide unique evidence concerning the effectiveness of telephone and direct mail campaigns and the synergy between the two. Our main findings, which have been widely disseminated at conferences and are forthcoming in Electoral Studies, provide key evidence that impersonal campaigns are effective and that the impact of different campaign interventions impact is cumulative. The evidence informs academic debates about the crucial role of mobilisation in understand electoral turnout. We also found that within a single election treatment effects do not vary significantly across locations, regardless of the levels of campaigning by the major political parties, political incumbency or marginality. This has hugely important implications for other similar studies carried out world-wide. What is suggests is that single location GOTV studies (of which there have been many) do not suffer the external validity problems that are commonly assumed. However, or research also implies that great care should be taken when comparing treatment effects across elections, and when comparing studies from high and low turnout areas. This of great importance to our understanding of previous and future GOTV studies.

We have also deposited data for re-analysis at the UK Date Archive which is crucial in experimental research so that other scholars can not only replicate the research but also build up evidence on treatment effects from a variety of contexts. This cannot be achieved by reference to published work alone due to problems of publication bias (whereby on the whole only positive findings are reported)


In the 2009 European Elections the telephone campaign, in isolation, initially had no significant effect on turnout. However, then Telephone campaign had a strong statistically significant effect (4%) in 2010, suggesting heterogeneity of effects across elections. Both estimates are consistent with a number of overseas studies that have shown calls made by volunteer phone banks have effects ranging from approximately zero to 8%. Direct mail did have a significant impact on voting, increasing turnout by a little over 1.5% in 2009 rising to just below 2.0% in 2010. Again, the smaller mail effect is consistent with studies carried out in the US.

We also examined the synergy of the two modes of intervention, subjecting one group to both telephone and mail treatments. In combination, the treatment is effective in persuading electors to vote, but the size of the treatment effect suggests nothing more than a simple additive impact. In focus groups, there was a clear difference between the double treatment group and the single treatment groups in their memory of the intervention.

The results for 2010 suggest that there is also some accumulation of effects over time, part of which may be due to habit effects and part due to the effect of repeat contact.

There is a substantial difference in the level of turnout between electors who are telephone accessible and those who are not (7% higher in the control group 2009 and 4% in 2010). This has important implications for comparing telephone treatment groups with control groups unless they are pre-screened for telephone accessibility.

The nationally representative sample allowed us to explore geographic variations in the effect of the treatment. There was little or no variation in treatment effects across places within an election. This has important implications for the external validity of GOTV field studies more generally, especially where single locations are used.

Overall, most of the treatments in the experiments had some effect suggesting that impersonal methods of voter mobilisation work. A summary of key findings is as follows:

• Treatment effects were greater at the 2010 General Election than in the 2009 European Elections

• Mail had small significant effect in both elections (larger in 2010)

• Telephone alone had no impact in 2009 but a large effect in 2010

• The combined treatment was statistically significant in both elections, but greater in 2010.

• Enhanced mail and telephone effects may result from follow up treatment but repeating the combined treatment was less effective in 2010 than the fresh combined treatment

• There is evidence for a simple additive effect of mail and telephone but not for synergy (an interactive effect);

• There is evidence of heterogeneity of treatment effects between elections, but there is little or no variation in treatment effects across places within an election.
Exploitation Route The research provides the most comprehensive evidence for the effectiveness of direct mail and telephone, both independently and in combination, as means of mobilising voters. This can be put to use by campaigners - whether these be political parties or governmental or non-governmental agencies who can use the findings to estimate the likely success rate of telephone and mail campaigns. This is being communicated to interested parties as set out in the impact report.

The findings also have scientific implications, most importantly in that we demonstrate that within any election, between place variations in treatment effects are minimal. However, between elections variations are large. This is of use to researchers designing future studies and drawing conclusions from existing studies. This is being communicated through academic conferences, workshops and publication in journals.
Sectors Government, Democracy and Justice
 
Title How different are direct mail and telephoning? : a GOTV randomised experiment in two elections 
Description This is an anonymised individual level dataset containing information neccessary to replicate analysis and re-analyse results. The data is anonymised and contains information on treatment and turnout. The first wave was carried out in 2009 (European Elections) and the second in 2010 (General Election). Treatment and control groups from 2009 were followed up in 2010, with some voters being re-contacted. Using a multistage sample design with stratification and clustering, we sampled 40,000 individuals from a commercial database based on electoral registers and telephone records. Eighty-three electoral wards were sampled randomly from 27 local authority districts. The sample was further stratified by household and restricted to one random person per household to avoid clustering, and to avoid some households receiving double treatments. This sample was further stratified according to telephone accessibility. In total 981 telephone accessible and 500 telephone inaccessible individuals were sampled in each local authority district, spread across 3 (or occasionally 4) sampled wards in the district. Each individual was randomly assigned to one of three treatment groups (telephone, mail or mail & telephone). Only after the randomisation was complete, any electors in the sample (treatment or control groups) that were not registered to vote were removed, leaving a sample (in 2009) of 25,366. At the General Election of 2010, we canvassed the sample again, but with the difference that we randomly allocated a portion of the control group to a new mail and telephone group. A proportion of the sample that was included in 2009 had left the electoral register in 2010 or had changed name/address details and was therefore excluded, leaving a sample of 21,984 in 2010. 
Type Of Material Database/Collection of data 
Year Produced 2012 
Provided To Others? Yes  
Impact this research was used in a number of academic papers published by the research team, and has been discussed with strategists in the Labour party in relation to the value of voter mobilization efforts. 
 
Description Do impersonal mobilisation methods work? : evidence from a get-out-the-vote experiment from the 2009 English European elections and 2010 general election 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Policymakers/politicians
Results and Impact Presentation to the Electoral Commission of research findings
Year(s) Of Engagement Activity 2012
 
Description Do impersonal mobilisation methods work? : evidence from a nationwide get-out-the-vote experiment in England 
Form Of Engagement Activity A formal working group, expert panel or dialogue
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Policymakers/politicians
Results and Impact A short circular flyer sent to non-academic partenrs and contacts summarising the research findings

Section not completed
Year(s) Of Engagement Activity 2012
 
Description Do impersonal mobilisation methods work? : evidence from a nationwide get-out-the-vote experiment in England 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach Regional
Primary Audience Policymakers/politicians
Results and Impact Government and Participation Research Workshop at the Department of Politics, University of Sheffield
Year(s) Of Engagement Activity 2012
 
Description Randomised controlled trials in the social sciences : methodological advances in experimental research annual conference 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Ed Fieldhouse presented a paper summarizing the 2009 finding at the ' Randomised controlled trials in the social sciences : methodological advances in experimental research annual conference' to a multidisciplinary audience in York on 20 September 2010
Year(s) Of Engagement Activity 2011
 
Description The effectiveness of impersonal mobilisation methods work? : evidence from a get-out-the-vote experiment from the 2009 English European elections 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Policymakers/politicians
Results and Impact University of Manchester Experiments Research Network third seminar
Year(s) Of Engagement Activity 2012