📣 Help Shape the Future of UKRI's Gateway to Research (GtR)

We're improving UKRI's Gateway to Research and are seeking your input! If you would be interested in being interviewed about the improvements we're making and to have your say about how we can make GtR more user-friendly, impactful, and effective for the Research and Innovation community, please email gateway@ukri.org.

Comparing in-person to self-completion interviews: the example of the 2021 European Social Survey in the UK.

Lead Research Organisation: City St George’s, University of London
Department Name: School of Social Sciences

Abstract

Face-to-face fieldwork is becoming more expensive in the UK whilst response rates have been declining. Many UK surveys have been considering whether a switch to using self-completion instead of interviewer-administered surveys offers a more efficient, digital, data collection future. This project funds the fielding of the 2021 ESS in the UK as a self-completion survey. That will be conducted in an overlapping period with the (already funded) face-to-face fieldwork. The dataset created will allow analysts to examine how similar the estimates from the self-completion survey are to those from the face-to-face survey.
 
Description This study involved administering the European Social Survey Round 10 questionnaire via a self-completion approach in Great Britain. The central objective was to compare the results from this self-completion approach with the ESS Round 10 face-to-face survey in Great Britain.

The study data set and documentation were deposited at the UK Data Archive in 2022 and were published by UKDA as part of its curated collection on 6 April 2023.

The data collection phase for the self-completion study ran from November 2021 to February 2022. This involved sending a postal invitation, two reminder letters, and a 'survey relaunch' letter to a random sample of GB addresses to ask them to complete a 50-minute online survey using the Round 10 European Social Survey questionnaire. A paper questionnaire was sent to non-responders with the second reminder.

All sampled addresses were sent a £5 unconditional cash incentive. The study also involved a conditional incentive experiment: those who responded were sent either £0, £5 or £10 depending on their allocation to incentive group based on the experiment included in the study. A further £5 unconditional incentive was sent to non-respondents in the relaunch phase.

An overall response rate of 36% was achieved, out of which 10% of the responses were completed on paper, while the rest completed the survey online. The total number of participants in this study was N=2908. This comfortably exceeded the predicted overall 25% for the study. The response rate varied somewhat between the conditional incentive groups: 34% where no conditional incentive was offered, 36% where £5 was offered, and 39% where £10 was offered. However, the response rate by incentive for the paper participants was stable across the three incentive groups (between 9.4% and 10.4%). The survey relaunch phase added around 3 percentage points to the response rate.

The achieved sample composition was very similar to that achieved for the ESS Round 10 face-to-face survey in Great Britain (this was carried out in 2021-22; with data available to download via the ESS Data Archive). This showed that across a range of indicators, including sex, age and years of education, the samples produced from the two surveys were similar.

Since the survey was completed online and on paper, we were able to closely investigate the breakdown of responses by mode and for the online responses, by device: 27% of the total sample completed the survey on paper; 23% used a mobile phone; 6% used a tablet; and about 43% used a computer.

Several publications have now been released, or are forthcoming, based on the study. This includes those listed below. Further work and publications are planned to compare results from the self-completion study with the face-to-face approach in Great Britain. The results to date have already made an important contribution to the development of ESS's future self-completion approach, which will be used as part of its mixed-mode 12th round (in 2025-26) and its first fully self-completion round (Round 13, 2027-28).

Comanaru, R., and Reece, N. (2023). "Next Birthday approach to sampling in an ESS experiment in Great Britain and beyond", ESRA Conference, Milan.

Comanaru, R., and Fitzgerald, R., 'The challenges of repeat measurement in times of the COVID-19 pandemic' in Ferrín, M., and Hanspeter, K. (eds) (forthcoming). How Europeans view an evaluate democracy revisited. Ten years later. Oxford: Oxford University Press.

Hanson, T., Fitzgerald, R., Ghirelli, N., and O'Muircheartaigh, S. (2022). "Delivering the European Social Survey During COVID-19: reflections and future implications", AAPOR Annual Conference, Chicago.

Lugtig, P. (forthcoming). "ESS round 10 mode experiments in Great Britain and Finland: Findings on mode effects". ESS Working Paper.

O'Muircheartaigh, S., Hanson, T., and Fitzgerald, R. (2023). "Challenges and successes of changing mode in a cross-national context: Developing a self-completion approach for the European Social Survey", ESRA Conference, Milan.

Reece, N., Comanaru, R., Fitzgerald, R., and Hanson, T. (2024). "Back to the future: the use of paper self-completion in the ESS mode transition", CSDI Workshop, Berlin.
Exploitation Route The results from this study will feed into the data collection mode switch for ESS in 2025 and 2027. They will help to guide users on the likely impact of the data collection switch on time series analysis. The results are feeding into the ESRC Survey Data Collection Methods Collaboration (Survey Futures).
Sectors Education

Healthcare

Government

Democracy and Justice

Other
 
Description The findings from this survey have been shared with the EU agency Eurofound along with other findings from the European Social Survey mode transition.
First Year Of Impact 2024
Sector Government, Democracy and Justice,Other
Impact Types Policy & public services
 
Title Social survey data collection 
Description The approach used in this survey - self completion data collection according to the European Social Survey methodology - has shown the high standards that can be achieved in surveys without field interviewers. 
Type Of Material Improvements to research infrastructure 
Year Produced 2024 
Provided To Others? Yes  
Impact The results of this study will be disseminated through the Survey Futures project, where a journal article is being prepared for submission in 2025. The results of this experiment have also contributed to the overall mode switch within the context of the European Social Survey. 
 
Title Self-Completion Study Based on the 2021 European Social Survey in Great Britain 
Description This dataset is the result of an experimental fielding of the Round 11 European Social Survey (ESS) as a self-completion (web and paper) survey in Great Britain. Data collection was carried out between November 2021 and March 2022. The total number of cases included in the data file is 2,908. This includes fully completed questionnaires and those where at least 75% of 'ask all' questions were answered. 2,116 responses were via web and 792 were on paper. The response rate was between 36% and 40%, depending on which assumption is applied regarding ineligible cases (see the technical report for further details). The experiment also included an incentive experiment, with different levels of conditional incentive randomly assigned to sample units. The incentive condition is flagged in the data file. It is expected that users may compare this data with the UK ESS Round 10 data (based on a face-to-face approach), which is expected to be released via the ESS Data Archive in early 2023. 
Type Of Material Database/Collection of data 
Year Produced 2023 
Provided To Others? Yes  
Impact Has been used in current and forthcoming publications. 
URL https://beta.ukdataservice.ac.uk/datacatalogue