Understanding survey response behaviour in a digital age: Mixed-device online surveys and mobile device use

Lead Research Organisation: University of Southampton
Department Name: Sch of Economic, Social & Political Sci

Abstract

We live in a digital age with technologies changing rapidly and entering all aspects of life including surveys. Surveys have also started adopting technologies including mobile devices for data collection. There is a big move in the direction of online data collection in the UK including the plan to collect 75% of household responses through the online mode of data collection in the UK 2021 Census. However, evidence is needed to demonstrate that an online data collection strategy will work in the UK. Unfortunately, so far not much is known about online data collection in the UK as the available datasets have small subsamples. This project addresses this gap and aims to analyse online response behaviour and data quality issues in the first available key ESRC large scale data resource as well as in another secondary data resource in the UK that are partially collected online with mobile devices as an available option for respondents - Understanding Society and Second Longitudinal Study of Young People in England (LSYPE2). These data will contain large enough samples to conduct advanced statistical modelling to address this project's research questions. One of the ESRC current main priority areas for investment is 'ways of being in a digital age' and this project addresses issues which belong to this area of priority by studying respondents' choice of devices used during completion of online surveys and their online survey behaviour.

Experience from other countries will be taken into account through the use of international advisors from the US and the Netherlands who will be part of the project's Advisory Committee. The project will also carry out some data analysis work in parallel with the Dutch partners from the Data Collection Innovation Centre. Results from the project will be compared to findings from the Netherlands to draw cross-country comparisons.

The findings from our project will be instrumental in better understanding participants' choices of devices used and response behaviour in mixed-device online surveys in the UK and in particular in informing best practice for the next UK Census in 2021. The project will also extend understanding of the data quality issues related to online mode of data collection with a special focus on respondents' use of different devices including mobile devices during survey completion process. These findings will inform knowledge that will lead to improvements in online data collection and as a result of this in data quality in the UK and other countries. The results will improve the understanding of who does and who does not respond in online surveys and which groups of respondents choose different devices and hence will inform the efforts of survey practitioners to increase response rates and to reduce survey errors and costs. The project will directly benefit statistical survey agencies and survey practitioners from a wide range of sectors, including government and the public sector, the private and voluntary sector, and market research companies. It will also advise researchers on how best to incorporate different devices data into advanced statistical models. The project will provide recommendations to survey practice organisations on which data need to be collected in the future in order to assess data quality of large scale surveys more effectively.

The project team will work closely together with a range of statistical agencies and market research companies, both nationally and internationally, to ensure the benefits of this research for survey practice and direct impact of the project findings.

Dissemination of the project findings will include peer-reviewed journal articles, presentations at seminars in the non-academic partner institutions and at national and international conferences as well as a short course and an international research symposium among other activities.

Planned Impact

Who will benefit?
We expect three groups to benefit from the research:
1. Academic researchers (social scientists, applied statisticians and survey researchers) and non-academic researchers (government researchers and survey researchers) with interest in the analysis of mixed-device online survey data. This group of users ranges from analysts who wish to learn about respondents' online survey behaviour to more advanced users who wish to consider the impacts of selection and data quality in online surveys on their substantive conclusions.

2. Data collection agencies and survey practitioners from commercial companies, survey research and market research organisations, government entities involved in data collection, and academic or non-profit survey organisations.


3. Survey sponsors and policy-makers. This group includes principle investigators of research projects, programme managers within government agencies and third sector organisations who issue large scale data collection and are responsible for and/or reliant on the quality of the statistics produced from these surveys.

How will they benefit?

1. Group 1 will obtain insights into the respondents' online survey behaviour and mobile device use as well as into selection and data quality issues associated with the online data collection. The project will further build capacity in UK Social Statistics through a short course, an international research symposium and through further advancing the skill set of the PI. Therefore it will contribute to the ESRC's long-term strategy of enhancing the quantitative skills of UK social scientists and will complement activities of other ESRC investments such as the NCRM and the Quantitative Methods Initiative. The research will also raise international awareness of the UK Social Science and its online survey research.

2. Group 2 will benefit from this research through a better understanding of issues associated with the online survey data collection. The in-depth analysis of online surveys will help to improve quality of future online survey data and will potentially improve the efficiency of data collection and reduce its costs. The research will benefit survey practitioners who are interested in increased survey response rates and in reduction of nonresponse in online surveys. The research will also provide guidance for design decisions for new surveys and will inform survey practice organisations on which different device data need to be collected in the future in order to assess data quality of large-scale surveys more effectively.

3. Group 3 often lacks the means to control the quality of survey data provided to them by data collection agencies. The policy-makers also draw on the Social Science research and are dependent on good quality of survey data. This group has to rely on a limited set of indicators (e.g., response rates) to evaluate the delivered product. This project will provide insights into the process of efficient online data collection and will allow outside control over the survey process.

What will be done to ensure that they benefit?

The project team has a strong record of engaging with non-academic as well as academic survey researchers and social scientists from a wide range of backgrounds. We will promote impact with non-academic partners through their direct involvement.

The proposed research will achieve impact through the completion of high-quality research which will be disseminated through research papers in leading journals, a policy brief with the main findings, seminar and conference presentations, an international research symposium, a short course and other activities. Further details of our dissemination strategy are outlined in the Pathways to Impact document.

This research will feed into academic and survey practice debates on future mixed-device online survey data collection and mixed-mode data collection strategies.

Publications

10 25 50
 
Description This project has made significant scientific contributions to a better understanding of data quality issues in mixed-device online surveys in the UK. There is a big move towards online data collection in the UK and globally, including the plan to collect 75% of household responses through online data collection in the UK 2021 Census. Also, UK Office for National Statistics (ONS) has a Social Surveys Transformation Programme with the plan to move the main Social Surveys online. Not much was known about online data collection in the UK at the beginning of this project so our key findings are very timely and help informing best practice for the Census 2021 and for transformations of UK social surveys towards online data collection and, therefore, crucially important for survey practice. One of the main focuses of our project was on comparison of data quality from smartphones to other devices such as PCs (desktops, laptops) and tablets. There are still concerns that smartphones produce lower quality data. These concerns caused that smartphones were blocked by some surveys in the past as a device for survey completion, later smartphone use was discouraged. Until now, some of the surveys still discourage smartphone use for survey completion. But are those claims regarding lower data quality produced by those using smartphones for survey completion reliable? The evidence was needed to demonstrate whether that was the case.

These project has addressed various aspects of data quality in mixed-device online surveys in the UK context.

The key findings from the project are:

1. Our results suggest that in the UK context younger people are more likely to use mobile devices for survey completion and older people are more likely to use PCs and laptops. Older people who are also widowed are more likely to use tablets for survey completion when compared to other devices. Also, females are more likely to use mobile devices for survey completion when compared to males. Employed respondents are more likely to use smartphones and tablets than PCs and laptops for survey completion in comparison to unemployed people. Those respondents who use the internet every day are more likely to use mobile devices for survey completion than PCs or laptops. Similarities in findings are observed in the UK and other countries and this provides evidence that our results might be generalisable to contexts in which this area is still underreshearched. These results are useful for tailored targeting of specific groups for survey participation and therefore might help increasing response rates and reducing costs of social surveys in the UK.

2. We have conducted an assessment of data quality in cross-sectional (LFS test data collected by the ONS, Understanding Society Innovation Panel and the main survey data) UK datasets. The main difference between the two settings assessed was that Understanding Society still used non-optimised questionnaires for mobile devices, whereas ONS used mobile-first design for the questionnaires. The results suggest that we should not be concerned about respondents using smartphones in future surveys even for longer surveys as data quality is not very different when compared to other devices. The results are similar in both contexts with non-optimised for smartphones questionnaires and with optimised questionnaires and these findings are very reassuring for survey practice and important for future designs of online surveys in the UK and globally.

3. As a part of the project we have also conducted an assessment of the first cross-national online probability-based panel (CRONOS) which collected data in Great Britain, Estonia and Slovenia. Data quality of the online probability-based panel was assessed through representativeness indicators (R-indicators). The results suggest that R-indicators are important tools to assess representativeness across time and between countries and should be used in survey practice to monitor and improve representativeness during data collection for online panels. This approach can be used by survey practice during the data collection process in the context of responsive survey designs to improve representativeness of the data collected within online panels.

4. Also, substantive results of this part of the project suggest that there are differences in representativeness over time in each country (Great Britain, Slovenia and Estonia) and also across three countries used for the analysis. The results also suggest that those with lower levels of education and those who are in the oldest age category and also in the youngest group in Great Britain context contribute more to the lack of representativeness in all three country contexts. These results are useful for targeting specific groups of the population in future online panels in the UK.
Exploitation Route Data collection organisations are undergoing paradigm shift and there is a big move towards online data collection in the UK and globally, including the plan to collect 75% of household responses through online data collection in the UK 2021 Census. Also, UK Office for National Statistics (ONS) has a Social Surveys Transformation Programme with the plan to move main Social Surveys online, for example, Labour Force Survey (LFS). Not much was known at the beginning of our project about online data collection in the UK and specifically about the quality of data collected through mixed-device online surveys so our key findings are very timely and help informing best practice for the Census 2021 and for transformations of UK social surveys towards online data collection and, therefore, very important for survey practice. Also, our results are useful for tailored targeting of specific groups for survey participation and therefore might help increasing response rates and reducing costs of social surveys in the UK.
A very successful international conference was organised by this project jointly with the ONS. It attracted non-academic audiences from across 14 countries which suggested timeliness of the project and usefulness of the project outcomes in the area. The conference initiated discussion across industries and demonstrated clear non-academic impact of the project.
The results of the project were widely disseminated to academic and non-academic audiences during the project. Also the results of the project initiated further funding application and establishment of a network of academic and non-academic partners working in the area of online data collection in social surveys in the UK and other countries.
Sectors Communities and Social Services/Policy,Digital/Communication/Information Technologies (including Software),Government, Democracy and Justice,Other

URL https://www.ncrm.ac.uk/research/datacollection/
 
Description There is a big move globally to online data collection. However, at the beginning of the project evidence was still needed to make online data collection work effectively and it was crucially important to test whether online mixed-device surveys in the UK produce reliable data which can be used for policy, financial and business decisions. Failure to do this could have resulted in large amounts of unreliable data and wasted resources. Our project has an important non-academic impact. The project produce first available in the UK evidence and recommendations for allowing mobile phones for survey completion in social surveys. The results of the analysis suggested that the data quality produced by smartphone users is not different from other device users (desktops, laptops and tablets) even in the context where the questionnaires were not optimised for mobile devices and small screens. These results were widely disseminated to non-academic users and are very important for the users, especially given that the data collection organisations are currently undergoing paradigm shift and many surveys have either already started collecting data online or are in the process of moving towards online data collection. Our project findings have been informing non-academic users such as survey practitioners in survey agencies and national statistical offices on respondents' online behaviour and the use of different devices in the UK context. Also, analysis of experiment data for the ONS Labour Force Survey was important for future decisions regarding online data collection in major Social Surveys in the UK. This non-academic impact was achieved through presenting our findings at conferences which are not only attended by academic colleagues but also by survey practitioners (for example, General Online Research (GOR) conference, ESRA conference, etc.), seminars the ONS, Royal Statistical Society (RSS) event which attracted many non-academic partners. Also, a very successful international conference which attracted 78 participants from 14 countries and from different industries which collect data online was a very important avenue for dissemination of findings in the area and for cross-industry discussion about the future of online data collection in social surveys. Non-academic impact was also achieved through direct collaboration with non-academic partners in the UK and abroad from data collection agencies such as Kantar Public, Ipsos-Mori, independent research institutes NatCen and GESIS (Germany) and national statistical offices such as the Office for National Statistics (ONS) and CBS Netherlands. The results produced by the project are used by various survey practice organisations in the UK and other countries when they design and implement new and improve existing online social surveys. Also another non-academic impact of the project was facilitation of the discussion and collaboration between various organisations involved into design and implementation of online social surveys in the UK and other countries. This collaboration helps to share experiences and good practices and also help to avoid duplication of efforts between organisations.
First Year Of Impact 2019
Sector Communities and Social Services/Policy,Digital/Communication/Information Technologies (including Software),Government, Democracy and Justice,Other
Impact Types Societal,Economic,Policy & public services

 
Description Short Course "Smartphones and Mobile Sensors in Online Surveys"
Geographic Reach Multiple continents/international 
Policy Influence Type Influenced training of practitioners or researchers
Impact The course helped improving skills of participants in the area of online data collection. High quality of data is important as lots of financial and policy decisions are based on data and therefore it is important that collected data is of high quality. This course presented best practice in the are of various types of online data collection.
 
Description Transitioning from Interviewer-Administered Surveys to Online Data Collection: Experiences, Challenges and Opportunities (GenPopWeb 2 Network)
Amount £47,666 (GBP)
Funding ID ES/V001051/1 
Organisation Economic and Social Research Council 
Sector Public
Country United Kingdom
Start 02/2020 
End 02/2021
 
Description Collaboration with GESIS 
Organisation Leibniz Association
Department Leibniz Institute for the Social Sciences
Country Germany 
Sector Academic/University 
PI Contribution We submitted a joint research proposal.
Collaborator Contribution Advisory board meetings. Attendance at the conference organised by the project.
Impact Submission of ORA proposal.
Start Year 2018
 
Description Collaboration with Kantar Public 
Organisation Kantar Group
Country United Kingdom 
Sector Private 
PI Contribution I led a publication in International Statistical Review journal where Tim Hanson from Kantar Public was a co-author.
Collaborator Contribution Advisory board meetings. Attendance at the conference organised by the project.
Impact Maslovskaya, O., Durrant, G., Smith P.W.F., Hanson, T., and Vilar, A. (2019) What do we know about mixed-device online surveys: Evidence from six UK surveys. International Statistical Review 87(2): 326-346. https://doi.org/10.1111/insr.12311|
Start Year 2018
 
Description Collaboration with NatCen 
Organisation National Centre for Social Research
Country United Kingdom 
Sector Charity/Non Profit 
PI Contribution We established closer links and created opportunities for joint projects and work
Collaborator Contribution NatCen attended advisory board meetings, helped to disseminate the results of our project in their institution. Gerry Nicolaas from NatCen was one of the keynote speakers at the conference organised as part of the project. She also presented at the RSS event organised as part of the project.
Impact Advisory board meetings. Keynote speech at our conference. Presentation at our RSS event. Successful ESRC project - ES/V001051/1. Application for another ESRC project.
Start Year 2018
 
Description Collaboration with Statistics Netherlands 
Organisation Statistics Netherlands
Country Netherlands 
Sector Public 
PI Contribution Joint application for a research grant
Collaborator Contribution Advisory board meetings. Advice between meetings.
Impact ORA application.
Start Year 2018
 
Description EES/NatCen/City University of London webinar 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Olga Maslovskaya gave an invited talk in survey methodology seminar series where she presented all findings from this project. It was at no costs to the project. 297 people registered for the webinar and obtained video recordings of the webinar and 130 participants were listening it live. The feedback was outstanding and I got many messages saying how important and useful the project outcomes are for their practice.
Year(s) Of Engagement Activity 2020
URL https://www.youtube.com/watch?v=pT2wygY_OeI
 
Description GOR conference presentation 2018 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Maslovskaya, O., Durrant, G., & Smith, P.W.F. (2018). Response quality in mixed-device online survey: Evidence from the Understanding Society Innovation Panel data. Paper presented at General Online Research (GOR) conference, Cologne, Germany, 28 February - 2 March 2018.
Year(s) Of Engagement Activity 2018
 
Description Invited seminar in the Office for National Statsitics presenting the results to Social Survey Transormation Team 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Olga Maslovskaya was invited to present results of the analysis from Labour Market Survey collected by the ONS to the ONS Social Survey Transformation team in Titchfield and Newport.
Year(s) Of Engagement Activity 2018
 
Description Organisation of the international conference on the future of data collection in social surveys 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We organised an international 2-day conference "The future of online data collection in social surveys" jointly with the ONS. The conference attracted 79 participants from academic and non-academic institutions from 14 countries. The event had a high non-academic impact as majority of participants were from non-academic organisations including government, private sector and market research organisations. We received an outstanding feedback and the participants commented on how useful and important the conference was. Various collaborations between academic and non-academic emerged as a result of the conference including GenPopWeb2 network funded by ESRC (ES/V001051/1).
Year(s) Of Engagement Activity 2019
URL http://www.ncrm.ac.uk/reserach/datacollection/
 
Description Presentation at American Association for Public Opinion Research (AAPOR) conference 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Maslovskaya, O., Durrant, G., Smith, P.W.F. (2019) Data quality in mixed-mode mixed-device general population UK social survey: Evidence from the Understanding Society Wave 8. Paper presented at American Association for Public Opinion Research (AAPOR) conference, Toronto, Canada, 16-19 May 2019.
Year(s) Of Engagement Activity 2019
 
Description Presentation at BigSurv20 conference (Big Data meets Survey Science) 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact It was a presentation at the international conference BigSurv20 which took place online this year. The session where I presented the talk was entitled 'Getting your estimates on point! Even more survey calibration approaches in the era of probability and nonprobability surveys'. As the whole conference was conducted in online format, the session attracted many more participants than it would usually attract in face-to-face format.
Year(s) Of Engagement Activity 2020
 
Description Presentation at British Society for Population Studies conference 2018 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Maslovskaya, O., Durrant, G., & Smith, P.W.F. (2018) Uptake and data quality in UK mixed-device online surveys. Paper presented at British Society for Population Studies (BSPS) Conference, University of Winchester, Winchester, UK, 12 September 2018.
Year(s) Of Engagement Activity 2018
 
Description Presentation at British Society of Population Studies (BSPS) conference 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Maslovskaya, O., Durrant, G., Smith, P.W.F. (2019) Data quality in mixed- mode mixed-device general population UK social surveys: Evidence from Understanding Society Wave 8. Paper presented at British Society for Population Studies (BSPS) conference. Cardiff University, Cardiff, 9-11 September 2019.
Year(s) Of Engagement Activity 2019
 
Description Presentation at Royal Staistical Society event 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Maslovskaya, O., Smith, P.W.F., Durrant, G. (2019) Data quality in mixed-device online surveys in the UK. Invited talk at Royal Statistical Society (RSS) event on the Future of Online Data Collection in Social Surveys, RSS, London, 21 November 2019.
Year(s) Of Engagement Activity 2019
 
Description Presentation at Understaning Society Scientific Conference 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Other audiences
Results and Impact Maslovskaya, O., Durrant, G., Smith, P.W.F. (2019) Data quality in mixed-mode mixed-device general population UK social survey: evidence from Understanding Society Wave 8. Paper presented at Understanding Society Scientific Conference. University of Essex, Colchester, 2-4 July 2019.
Year(s) Of Engagement Activity 2019
 
Description Presentaton at ESRC Researh Methods Festival 2018 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Maslovskaya, O., Durrant, G., & Smith, P.W.F. (2018) Response quality in mixed-device online survey: Evidence from the Understanding Society Innovation Panel Wave 9 data. Paper presented at the ESRC Research Methods Festival, University of Bath, Bath, UK, 03 July 2018.
Year(s) Of Engagement Activity 2018
 
Description Royal Statistical Society event on the future of online data collection in social surveys 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact We have organised an event at the Royal Statistical Society free of charge to the project but directly linked to it. The event was on the future of online data collection and attracted survey practitioners from a wide range of companies mainly from London. There were three presentation (ours, ONS's and NatCen's) which discussed best practice in transitioning to online data collection. At the end of the event we had a panel discussion where participants could ask questions on variety of topics related to online data collection.
Year(s) Of Engagement Activity 2019
URL https://rss.org.uk/news-publication/news-publications/2020/section-group-reports/social-statistics-s...
 
Description Special issue on the future of online data collection in social surveys in Journal of Royal Statistical Society Series A 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact We are currently finalising a special issue in a high-quality peer-reviewed journal which was one of the outputs of the project. This special issue brought together important publications in the area of online data collection in socials surveys and will be beneficial for academics, students and survey practitioners.
Year(s) Of Engagement Activity 2021
 
Description Stakeholder engagement at Statistics New Zealand 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Professor Peter WF Smith visited Statistics New Zealand and had a discussion with colleagues from there where he discussed our project and our results.
Year(s) Of Engagement Activity 2018
 
Description Three presentations at European Survey Research Association conference 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Maslovskaya, O., Lugtig, P., Durrant, G. (2019) Investigation of Non-Response Bias and Representativeness in the First Cross-National Probability Based Online Panel (CRONOS). Paper presented at European Survey Research Association (ESRA) conference, Zagreb, Croatia, 15-19 July 2019.

Maslovskaya, O., Durrant, G., Smith, P.W.F. (2019) Data Quality in Mixed-Mode Mixed-Device General Population UK Social Survey: Evidence from the Understanding Society Wave 8. Paper presented at European Survey Research Association (ESRA) conference, Zagreb, Croatia, 15-19 July 2019.

Phelps, A., Maslovskaya, O., Durrant, G. (2019) Are Smartphones Really the 'Bad Guys'? Dispelling Smartphone Data Collection Myths: Evidence from a Large Random Probability Online UK Survey Run by ONS. Paper presented at European Survey Research Association (ESRA) conference, Zagreb, Croatia, 15-19 July 2019.
Year(s) Of Engagement Activity 2019
 
Description Two Presentations at GOR conference 2019 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other audiences
Results and Impact Maslovskaya, O., Durrant, G., Smith, P.W.F. (2019) Data quality in mixed-mode mixed-device general population UK social survey: Evidence from the Understanding Society Wave 8. Paper presented at General Online Research (GOR) conference, TH Koln University of Applied Sciences, Cologne, Germany, 6 - 8 March 2019.

Maslovskaya, O., Durrant, G., Smith, P.W.F. (2019) Dispelling smartphone data collection myths: Update and data quality in the UK Office for National Statistics (ONS) large random probability mixed-device online survey experiments. Paper presented at General Online Research (GOR) conference, TH Koln University of Applied Sciences, Cologne, Germany, 6 - 8 March 2019.
Year(s) Of Engagement Activity 2019