Transitioning from Interviewer-Administered Surveys to Online Data Collection: Experiences, Challenges and Opportunities (GenPopWeb 2 Network)
Lead Research Organisation:
University of Southampton
Department Name: Sch of Economic, Social & Political Sci
Abstract
Social survey data underpin many important public policy and economic decisions, and high quality data are crucial. Survey research in the UK and globally is currently undergoing a paradigm shift in data collection away from interviewer-administered surveys to online data collection. Social surveys and censuses of the general population are experiencing major transformations, with mixed-mode designs incorporating web and online only panels becoming more commonly used. This paradigm shift is driven by ongoing issues affecting interviewer-administered surveys, such as declining response rates, increasing survey costs, as well as societal trends towards greater technology use, in particular use of smartphones in daily life. The main aim of this network will be to address the challenges and opportunities of transitioning from interviewer-administered to online data collection through sharing knowledge and producing recommendations for high quality online data collection.
Globally and within the UK, there is a big move in the direction of online data collection in a mixed-mode context for censuses and surveys. For example, the 2015 Japanese census as well as the 2016 Canadian Census have moved to primarily online administration. In the UK the Office for National Statistics (ONS) is aiming to collect 75% of its 2021 Census household responses online. A number of existing surveys have also used a mixed-mode approach. Longitudinal surveys are at the forefront of this trend with several major surveys in the UK and internationally already taking, or trialling, this approach. Mixed-mode has also been used for cross-sectional surveys and there is strong interest from government departments for adopting these approaches for existing as well as new surveys. Another major trend has been towards the establishment in many different countries of probability-based web panels.
There is a growing body of experience and evidence about how online surveys should be designed and implemented. However, this is an emerging and fast-moving area of survey practice and ongoing technological changes present new challenges and opportunities. It is of crucial importance to share knowledge and best practice between organisations which design and implement online surveys, as well as data users. This is to ensure that social surveys are designed optimally for collecting high quality data, and that knowledge and experiences are shared, and these are the aims of this network. Different organisations such as academic institutions, data collection organisations, independent research institutes, governmental organisations and other organisations collecting data online should act as a joint front to share knowledge and best practice in order to facilitate smooth transitions between interviewer-administered surveys and online data collection, to improve already established online surveys or mixed-mode designs, to avoid duplicated efforts and therefore to help reduce costs while investigating and implementing the best ways forward in the world of fast-moving technology.
This project proposes setting up a network of partners (GenPopWeb2) which will facilitate this cross-industry collaboration and knowledge exchange between different stakeholders about the optimal designs of general population web surveys. The GenPopWeb2 Network will build on the results of the first GenPopWeb network funded by the ESRC through the NCRM. Since 2014, there has been a lot of research and new developments in this area, and the trend towards increasing use of online data collection has continued. This network aims to address the main challenges and gaps in knowledge which are of crucial importance for transitioning to online data collection in the UK and internationally and to enable sharing of knowledge across industries.
Globally and within the UK, there is a big move in the direction of online data collection in a mixed-mode context for censuses and surveys. For example, the 2015 Japanese census as well as the 2016 Canadian Census have moved to primarily online administration. In the UK the Office for National Statistics (ONS) is aiming to collect 75% of its 2021 Census household responses online. A number of existing surveys have also used a mixed-mode approach. Longitudinal surveys are at the forefront of this trend with several major surveys in the UK and internationally already taking, or trialling, this approach. Mixed-mode has also been used for cross-sectional surveys and there is strong interest from government departments for adopting these approaches for existing as well as new surveys. Another major trend has been towards the establishment in many different countries of probability-based web panels.
There is a growing body of experience and evidence about how online surveys should be designed and implemented. However, this is an emerging and fast-moving area of survey practice and ongoing technological changes present new challenges and opportunities. It is of crucial importance to share knowledge and best practice between organisations which design and implement online surveys, as well as data users. This is to ensure that social surveys are designed optimally for collecting high quality data, and that knowledge and experiences are shared, and these are the aims of this network. Different organisations such as academic institutions, data collection organisations, independent research institutes, governmental organisations and other organisations collecting data online should act as a joint front to share knowledge and best practice in order to facilitate smooth transitions between interviewer-administered surveys and online data collection, to improve already established online surveys or mixed-mode designs, to avoid duplicated efforts and therefore to help reduce costs while investigating and implementing the best ways forward in the world of fast-moving technology.
This project proposes setting up a network of partners (GenPopWeb2) which will facilitate this cross-industry collaboration and knowledge exchange between different stakeholders about the optimal designs of general population web surveys. The GenPopWeb2 Network will build on the results of the first GenPopWeb network funded by the ESRC through the NCRM. Since 2014, there has been a lot of research and new developments in this area, and the trend towards increasing use of online data collection has continued. This network aims to address the main challenges and gaps in knowledge which are of crucial importance for transitioning to online data collection in the UK and internationally and to enable sharing of knowledge across industries.
Planned Impact
Data collection organisations are undergoing a paradigm shift and moving towards online data collection.
The network and its final report will enable the collation and sharing of scientific knowledge and best practice in the area of online data collection. Also, as many organisations are conducting various experiments in the area, collating and sharing this knowledge through the web database will help to reduce duplication of efforts.
The report will identify what best practice is and therefore will ease the process of setting up a new online survey or transitioning from interviewer-administered to online data collection, as well as improving existing online data collection. It will therefore save costs and improve the quality of data collected online. Cross-industry collaboration and the final report with recommendations and agreed principles will help to deliver non-academic impact in survey practice.
The network will contribute to one of the core goals of the NCRM which is sharing good practice by bringing together academics and partners from outside academia to ensure and enable non-academic impact.
Network events will ensure collaboration and discussions across sectors involved in design and implementation of online surveys in the UK and internationally.
The network and its final report will enable the collation and sharing of scientific knowledge and best practice in the area of online data collection. Also, as many organisations are conducting various experiments in the area, collating and sharing this knowledge through the web database will help to reduce duplication of efforts.
The report will identify what best practice is and therefore will ease the process of setting up a new online survey or transitioning from interviewer-administered to online data collection, as well as improving existing online data collection. It will therefore save costs and improve the quality of data collected online. Cross-industry collaboration and the final report with recommendations and agreed principles will help to deliver non-academic impact in survey practice.
The network will contribute to one of the core goals of the NCRM which is sharing good practice by bringing together academics and partners from outside academia to ensure and enable non-academic impact.
Network events will ensure collaboration and discussions across sectors involved in design and implementation of online surveys in the UK and internationally.
| Description | The main aims of the project were: To create a network which will collate and share knowledge on how to effectively conduct online data collection in probability-based cross-sectional and longitudinal social surveys of the general population and sub-populations, including probability-based online panels, push-to-web surveys, and different mixed-mode designs that include online data collection. We have created a successful (and still functioning (beyond the project)) network of academic and non-academic partners in the UK and abroad which is active and collaborates in the area of transitioning to online data collection. We organised 7 highly attended public events where specific issues associated with online data collection are discussed and best practice in specific areas of online data collection is identified and shared. We also organised a closed meeting with experts which laid foundation for the important report on adjustments for mode effects. This is a very fast moving area and it is important to share knowledge and experience between organisations to achieve higher data quality in different social surveys and the network facilitate discussions and sharing the knowledge. We also identify the agenda of research required in the area and were successful in obtaining follow-up funding from the ESRC (Survey Data Collection Network (SDC-Net). We have organised an event which helped identify the best practice in the area of within-household selection methods in online surveys. The main problem in the UK is that we do not have register of individuals and address based files are used as a sampling frame which creates challenges. There is an issue of selection of individuals for social surveys in households and the event address this very timely issue and identified advantages and disadvantages of different methods available in survey methodology which will be beneficial for survey practitioners. We have also organised three workshops on various aspects of complex measures in online surveys: measuring cognition, collecting data linkage consents and occupational coding. The slides from all these events are available via GenPopWeb2 website and summary from each events will be included to the final report and will help identify priority areas for future research. We have also organised two events which addressed issues of transitioning long questionnaires to online data collection and also online questionnaire design, development and testing. The final event was on covid-19 and transitioning to online data collection which laid foundation for our application for further funding and the key finding was that not all social surveys will be moving to online data collection, online surveys are important and have their role but face-to-face surveys are here to stay and it is important to understand issues associated not only with transitioning with online data collection but also with return of face-to-face interviewing post covid pandemic. We are also working on three literature reviews in the key areas of transitioning to online data collection which we identified. The literature reviews look not only at published papers (it takes long time until papers appear in journals) but also at internal reports which were produced by different data collection organisations. Once finalised, these information will be very valuable for non-academic partners as will ensure that there are no duplicated efforts and that there is awareness of what was tried and what worked across different organisations and what is the current best practice. Covid pandemic slowed the progress with those but they should become available to publish shortly. The team is currently working on the final report from GenPopWeb2 project which will summarise all main messages and key findings and will be shared widely with academic and non-academic partners. The report will identify high priority areas for research in the area of online data collection and also implications for survey practice. |
| Exploitation Route | The network investigates the issues and challenges associated with online data collection and collages existing evidence and share knowledge and best practice in the area. The area is very fast-moving and published evidence might not be available yet but of high demand. The network facilitated knowledge and experience sharing and initiated cross industry discussions of best practices. All events were very well-attended and the need to move them to online format due to the pandemic helped to attract wider audiences and allowed wide international reach. We are producing reports and will share them with wide audiences. The reports will contain recommendations and agreed principles. These produced by network documents will be used across industries and will help to improve quality of online data in social surveys. One of the main contributions of the documents produced will be reports of best practice which could be used by academic and non-academic organisations during design and implementation of online data collection. The documents will also benefit international non-academic partners as they will be able to share the best practices within their institutions. Previous events in the area highlighted the strong demand for documents like this. The network ensured that efforts across different organisations and industries were not duplicated and different network partners have access to all resources generated by the network activities which will allow for easier design and implementation of online data collection in social surveys. For example, the draft of the report on adjustments for mode effects is in high demand and was already shared with some of the partners before being finalised and published on the project website. |
| Sectors | Government, Democracy and Justice,Other |
| URL | https://www.ncrm.ac.uk/research/genpopweb2/ |
| Description | The events we organise during the life of the grant discussed timely topics related to transitioning from interviewer-administered surveys to online data collection. This process of transitioning was sped up by covid-19 pandemic. However, there were many issues which required special attention and understanding when online social surveys were designed and implemented. Not many publications existed in this very fast-moving area, therefore, collaboration between academic and non-academic partners was crucial. The events we organised brought together academic and non-academic audiences which helped to identify the best practice in various areas of online data collection and improved knowledge exchange between different organisations. They also increased awareness of work being conducted in different parts of the industry and allowed participants to learn about best practice for design and implementation of online social surveys. The reports produced and the presentation slides made available are used across different organisations and different stakeholders and will continue helping to improve quality of online data in social surveys in the UK and beyond. These materials will ensure that efforts across different organisations and industries are not duplicated and different network partners have access to all resources generated by network activities (all available via project's website) which will allow for easier design and implementation of online data collection in social surveys and as a result of it, better quality data for policy and other decisions. The importance of the project was recognised and the follow-up funding was awarded to the team via NCRM for the project "The impact of COVID-19 on survey data collection methods in the social sciences". These two projects laid a lot of ground work for the application for the Survey Data Collection Methods Collaboration call. |
| First Year Of Impact | 2020 |
| Sector | Government, Democracy and Justice,Other |
| Impact Types | Societal,Economic,Policy & public services |
| Description | Collaboration with NatCen |
| Organisation | National Centre for Social Research |
| Country | United Kingdom |
| Sector | Charity/Non Profit |
| PI Contribution | We established closer links and created opportunities for joint projects and work |
| Collaborator Contribution | NatCen attended advisory board meetings, helped to disseminate the results of our project in their institution. Gerry Nicolaas from NatCen was one of the keynote speakers at the conference organised as part of the project. She also presented at the RSS event organised as part of the project. |
| Impact | Advisory board meetings. Keynote speech at our conference. Presentation at our RSS event. Successful ESRC project - ES/V001051/1. Application for another ESRC project. |
| Start Year | 2018 |
| Description | Complex measures in online surveys: data linkage consents (event 2) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | We organised a very important event in the area of complex measures in online surveys and invited experts in the field to discuss issues associated with transitioning to online data collection. This event specifically focused on collecting data linkage consents in online surveys. 82 people registered for the event and 62 people attended the event. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.ncrm.ac.uk/research/genpopweb2/ |
| Description | Complex measures in online surveys: measuring cognition (event 1) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | We organised a very important event in the area of complex measures in online surveys and invited experts in the field to discuss issues associated with transitioning to online data collection. This event specifically focused on measuring cognition in online surveys. 123 people registered fort he event and 90 people attended the event. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.ncrm.ac.uk/research/genpopweb2/ |
| Description | Complex measures in online surveys: occupational coding (event 3) |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | We organised a very important event in the area of complex measures in online surveys and invited experts in the field to discuss issues associated with transitioning to online data collection. This event specifically focused on occupational coding in online surveys. 72 people registered for the event and 60 people attended the event. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.ncrm.ac.uk/research/genpopweb2/ |
| Description | Conference presentation - GOR 2022 conference in Berlin - Barriers to transitioning to online data collection in social surveys: Findings from GenPopWeb2 project |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Presentation give at a GOR conference to report on the findings from this project |
| Year(s) Of Engagement Activity | 2022 |
| Description | Conference presentation - WAPOR 2022 conference in Dubai - Barriers to transitioning to online data collection in social surveys: Findings from GenPopWeb2 project |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | The main findings from GenPopWeb2 project were presented at the WAPOR 2022 international conference |
| Year(s) Of Engagement Activity | 2022 |
| Description | Covid-19 and transitioning to online data collection in social surveys |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | This was a very important event which created a base for the next project and creation of SDC-Net network. 156 people registered for the event and 102 attended the event in real time. The event attracted large number of academic and non-academic partners in the UK and internationally. There were 5 presentations: 1. Food and You 2: moving FSA's flagship survey online; 2. Learning from transitioning face-to-face studies to push-to-web; 3. Health Survey for England: Exploring alternative mode; 4. Measuring crime during the coronavirus (Covid-19) pandemic: Crime Survey for England and Wales (CSEW); 5. The cross-national challenge: Moving (some) European Social Survey Countries to "web first" during the pandemic. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.ncrm.ac.uk/research/genpopweb2/ |
| Description | Designing and delivering online surveys: Transitioning long questionnaires |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | This event addressed issues associated with transitioning long questionnaires to online data collection. 182 people registered for the event and 132 people attended the event. There were three presentations by academic and non-academic partners: 1. Developing self-completion instruments for the European Social Survey; 2. Moving a long survey online: Problems and some potential solutions; 3. Transforming social surveys in ONS: Opportunities and challenges. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.ncrm.ac.uk/research/genpopweb2/ |
| Description | Meeting with experts to discuss adjustments for mode effects |
| Form Of Engagement Activity | A formal working group, expert panel or dialogue |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | A virtual meeting with experts took place in September 2020 to discuss adjustments for mode effects which is a very important and under-researched area in mixed-mode surveys. A report is being prepared and will be shared with a range of audiences via the network's mailing list and other channels. |
| Year(s) Of Engagement Activity | 2020 |
| Description | Online questionnaire design, development and testing |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | This was an important event organised online which attracted many academic and non-academic partners. 66 people registered for the event and 61 people attended this online event. The topic of the event was Online questionnaire design, development and testing. This topic was of crucial importance during the covid pandemic. There were three presentations from non-academic partners: 1. Creating an accessible survey platform; 2. Online questionnaire testing: How has the pandemic changed our approach at NatCen? 3. Putting respondents first: user centred design approaches at ONS. |
| Year(s) Of Engagement Activity | 2021 |
| URL | https://www.ncrm.ac.uk/research/genpopweb2/ |
| Description | Webinar: Within-household selection methods for probability web surveys |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | An important online workshop on within-household selection methods for probability web surveys brought together experts in the area to discuss advantages and limitations of different methods. This discussion was very timely and important for transitioning to online data collection which was sped up by COVID-19 pandemic. 304 people registered for the event, 305 people viewed the recording on YouTube channel and 144 people attended the event online. |
| Year(s) Of Engagement Activity | 2020 |
| URL | https://www.ncrm.ac.uk/research/genpopweb2/ |