Survey data quality in panel surveys: trade-offs between nonresponse and measurement errors

Lead Research Organisation: University of Essex
Department Name: Inst for Social and Economic Research

Abstract

This project aims to improve data quality in panel surveys, while at the same time lowering survey costs. Panel surveys involve groups of people or households that are followed over time. Typically, they answer questions annually, about their life, attitudes and behaviours. From the resulting data from these surveys, we can learn a lot about life in Britain. This is especially the case when a survey documents the life of a representative sample of people, as is the case in the British Household Panel Study (BHPS). From the BHPS we can learn how many people are unemployed every year, who they are, and how and when they find a job again for example. A return to employment is in turn likely to affect household income, and family wellbeing. In short, panel studies have the power to enrich our knowledge about how people's lives change over time, at the individual level, and in Britain as a whole.

This only is the case however when the measurement of all topics of interest are without error. There are two sources of error that threaten to make panel data invalid and unreliable. First, nonresponse among specific respondents, and second errors in measurement of the topic of interest using survey questions. Nonresponse is particularly likely to lead to biased data when specific groups of respondents are likely to drop out. This might be exactly the case for people who for example move because they have found a new job. Earlier research has suggested that people are less likely to participate in a survey when they are in the process of moving house. The reason for moving is however often a change of job. This in turn leads the survey data to under represent people who change jobs; one of the topics of interest in panel surveys.

Similarly, measurement error might also occur for specific groups of people. For example, people with lower cognitive abilities have more difficulty understanding questions, and remembering any information required to give a correct answer. This in turn might lead to biased data when people are asked about their knowledge and opinions of policy issues.

Survey methodologists worry that errors due to nonresponse and measurement interact. Some reasons for nonresponse might at the same time also be a reason for reporting with more measurement error. Lower cognitive abilities, complex income compositions, or language difficulties are among the reasons for both nonresponse and measurement error.
In this proposal, trade-offs and common causes for both nonresponse error and measurement error are studied.

The first part of this project is devoted to developing models to separate the two types of error; we normally have no information on measurement error for those people that do not participate, so statistical models are necessary to overcome this problem. The second part studies common causes of the two error sources, and identifies which respondent characteristics are responsible for them. The final part studies the trade-offs between measurement error and nonresponse. Understanding the trade-off better will enable researchers to compare the nature and size of both errors, and make better informed decisions in trying to limit survey errors. This in turn will make future panel studies more cost-effective and provide higher data quality.

Planned Impact

Longitudinal survey data are the source for many social scientists, economists, and policy makers who study or take policy-decisions regarding poverty, income and well-being. In the UK, research based on longitudinal surveys such as the British Household Panel Survey (BHPS), the Birth Cohort Studies and the English Longitudinal Study of Aging (ELSA) has over many years had a huge impact on policy debates and policy decisions. This study should improve the quality of such data and of the research based upon those data.

Specifically, any lessons regarding improved analysis approaches that better take into account measurement error, non-response error and the interaction between them will be of use to analysts across the broad range of disciplines that use longitudinal survey data, including sociology, economics, geography, psychology, politics, health sciences and demography. Although this includes research in academia, it also includes researchers in market-research and government agencies; for example the home office, and Office for National Statistics.

Market researchers and survey agencies perform much of the fieldwork for companies, government agencies and academia. Market researchers have over the years faced increasing pressure on the quality and costs of doing surveys. Because of increasing nonresponse rates and the higher survey costs associated with it, many market researchers have left traditional sampling methods, and have instead opted for using non-probability samples in the form of volunteer opt-in panels. Even within those panels however, nonresponse among panellists is a large problem. Market researchers are also worried about the answering behaviour of respondents in their panel surveys. Respondents often do not take the time and effort to complete surveys seriously, and instead opt for a strategy that takes the least effort against the least amount of time, resulting in strategies as "nondifferentiation", " straightlining" and "speeding".
Such undesirable answer strategies are often dealt with using 'ad hoc' methods. The guidelines that this project will produce will allow market researchers to easily study the extent of nonresponse and measurement error in their panel studies. The guidelines will enable market researchers to decide whether to convince unactive members to become active again, and whether measurement error really is as large as a problem as many suspect it is. Finally, market researchers will be able to use the guidelines to whether nonresponse or measurement error is the largest problem in their survey. The statistical models that this project will use are rather complicate for some people without a solid statistical background, The guidelines will be practical however so that everyone can use and implement them. If any statistical models are necessary, specific programming syntax for all major statistical packages will be added to the guideline as an appendix. To this end, we will actively work together with the Market Research Society (MRS) in designing and writing down these guidelines. We will also collaborate with the MRS to disseminate the guidelines to market researchers and survey agencies.

Government agencies will benefit from this project in two ways. First, by the improved data quality of the panel surveys it uses for decision making, leading to better informed policy-decisions. Second, by the lower costs of panel surveys, meaning that funding costs of future panel surveys will be lower.

It should be noted that academic researchers will also benefit from improvements that are implemented by the non-academic sectors, notably government and private sector survey agencies. Academics use a lot of data collected outside of academia so the impacts on non-academic audiences (see "Pathways to Impact") are highly relevant for academics too.
 
Description - Only specific forms of attrition in a survey can be predicted. Most notably attrition due to refusal by the respondent.
- Respondents who drop out from surveys have a different profile from respondents who continue to participate in panel surveys. The most notable differences between the two types are not merely demographic, but also psychographic. More conscientious and introvert people are more likely to become 'loyal' panel members.
- Item missings, and lower cooperation rates by the interviewer are good predictors of later dropout from a panel study.
- A theoretical model for understanding trade-offs between nonresponse and measurement error has been developed and described.
- A statistical model to estimate the size of both nonresponse and measurement error has been developed.
Exploitation Route Findings can be used to inform decision-making in so-called adaptive survey designs. These design gain a lot of interest in recent years from survey methodologists and survey practioners. The project has found for example that specific forms of attrition (due to noncontact, refusal etc.) can be predicted to a different extent. Refusals are the one cause for attrition that can be predicted fairly well. Using covariates, I have identified who will attrite. This finding can be used to target those likely refusers, and try to prevent them attriting, or perhaps to re-recruit them into the survey.
Sectors Government, Democracy and Justice,Other

URL http://www.peterlugtig.com
 
Description Findings from my research have been used by survey practitioners. Specifically the statistical offices in the Netherlands and Sweden, to inform the design of (future) longitudinal studies.
First Year Of Impact 2013
Sector Government, Democracy and Justice,Other
Impact Types Policy & public services

 
Description Statistics Netherlands: advisory and collaborative work
Geographic Reach National 
Policy Influence Type Participation in a advisory committee
Impact Participated in consultation of Statistics Netherlands on attrition in several longitudinal surveys they are running. As of 2016, we are setting up a formal research network to facilitate consultation on survey methodology, and collaborative research on the prevention of attrition
 
Description WIn project 
Organisation Statistics Netherlands
Country Netherlands 
Sector Public 
PI Contribution Set up a collaborating rpoject with statistics netherlands investigating the use of smartphone apps as a way to build an ongoing longtitudinal survey in the Netherlands. Together we have got funding to build an app (2 FtE), test it (1 FTE), and make the data usable for official statistics (1 FTE)
Collaborator Contribution see above
Impact Bais, F., Schouten, J.G., Lugtig, P., Toepoel, V., Arends-Toth, J., Douhou, S., Kieruj, N., Morren, M. and Vis, C. (2017) Can survey item characteristics relevant to mode-specific measurement error be coded reliably? Sociological Methods & Research Lugtig, P, Toepoel, V., Haan, M. and Schouten, J.G. (2017) Het Waarneem Innovatie Netwerk. Stator, 3, 16-19
Start Year 2016
 
Description 2 presentations at TSE conference 2015 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact One invited presentation on ' Total survey error for longitudinal surveys' (with peter Lynn)
One contributed presentation on 'a common metric for nonresponse and measurement error'.
Sparked discussion at conference. Materials of both chapters are now used in course at University of Michigan on total survey error
Year(s) Of Engagement Activity 2015
 
Description Blog post about item-missings and relation to attrition 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact had requests for paper that will be based on blog post

not aware of any impact
Year(s) Of Engagement Activity 2014
URL http://www.peterlugtig.com/2014/04/are-item-missings-related-to-later.html
 
Description ESRA conference, Reykjavik 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Gave three presentations at ESRA conference. This is the largest conference for survey researchers in Europe. Based on presentations Ireceived some feedback on presentations which helped in improving papers. Two future collaborations with industry in the field of attrition on smartphone panel surveys resulted from this.
Year(s) Of Engagement Activity 2015
 
Description Invited talk at GESIS summerschool about survey errors 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Invited lecture on importance of understanding survey errors to participants of summer school in survey methods

led to a few students choosing to do a specific methods-related part in their ph.d. thesis
Year(s) Of Engagement Activity 2013
 
Description Nonresponse workshop 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Discussion meeting of about 50 people from academia and national statistical offices, with the goal to exchange information on survey practice and theory. Led to invited talks at institutes, and potential collaboration with Statistics Sweden

collaboration with statistical office of Sweden on designing longitudinal surveys
Year(s) Of Engagement Activity 2013,2014
URL http://www.nonresponse.org
 
Description blog post: do respondents become sloppy before attrition 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact had discussion on forum about different ways to model attrition

not aware of impact
Year(s) Of Engagement Activity 2014
URL http://www.peterlugtig.com/2014/03/do-respondents-become-sloppy-before.html
 
Description blogpost: adaptive panel designs 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact blogpost describing how the findings from ESRC project on the trade-off between measurement and nonresponse error can be used to inform adaptive designs, where data collection protocols are adapted to the inidividual and fieldwork data.
Year(s) Of Engagement Activity 2015
URL http://www.peterlugtig.com/2015/05/adaptive-designs-4-ways-to-improve.html
 
Description blogpost: longitudinal interview outcome data reduction 
Form Of Engagement Activity A magazine, newsletter or online publication
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact had discussion at weblog about using Latent Classes or sequence analysis to model longitdinal interview outcome data.

not aware
Year(s) Of Engagement Activity 2014
 
Description blogpost: the tarditional web survey is dead 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Blogpost on how web surveys are changing due to increased use of smartphones and tablets. Reached about 2.000 people through twitter, 500 people on blog directly. Received 10 re-tweets.
Year(s) Of Engagement Activity 2016
URL http://www.peterlugtig.com/2016/02/the-old-web-survey-is-dead.html
 
Description blogpost: why panel surveys need to go adaptive 
Form Of Engagement Activity Engagement focused website, blog or social media channel
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Postgraduate students
Results and Impact Blogpost in which I describe the increasing extent of attrition in longitudinal surveys in Europe, and what should be done to prevent this. Blogpost generated debate on weblog, and subsequent conferences.
Year(s) Of Engagement Activity 2015
URL http://www.peterlugtig.com/2015/01/why-panel-surveys-need-to-go-adaptive.html
 
Description invited talk at Institute of Education, London on January 21, 2014 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach Local
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact 50 people from Institute of Education attended my talk on attrition from longitudinal surveys

Led to future collaboration between me and Institute of Education
Year(s) Of Engagement Activity 2014
 
Description participation at AAPOR conference 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Professional Practitioners
Results and Impact Talks at 2013 and 2014 conferences of the American Association for Public Opinion Research about ESRC projects and research that was part of project. Had 1 talk in 2013, and 3 talks at conference, which is most high-profile conference in the field of survey methods

Had requests for paper, and collaborations.
Year(s) Of Engagement Activity 2013,2014
URL http://www.aapor.org
 
Description participation in Panel Survey Methods Workshop 
Form Of Engagement Activity Participation in an activity, workshop or similar
Part Of Official Scheme? No
Geographic Reach International
Primary Audience Other academic audiences (collaborators, peers etc.)
Results and Impact 50 people attended my talk and talks of others at conference, which spearked discussion about design of longitudinal surveys

No notable impacts, apart from information sharing and discussion. Led to ideas for related future research
Year(s) Of Engagement Activity 2014
URL http://www.panelsurveymethods.wordpress.com
 
Description presentation at GESIS/Mannheim seminar series 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Professional Practitioners
Results and Impact Gave a presentation on ESRC project findings in a joint seminar series on ' research methods' of GESIS and the University of Mannheim, Germany. 1 april 2015
Year(s) Of Engagement Activity 2015
 
Description talk at Statistics Netherlands 
Form Of Engagement Activity A talk or presentation
Part Of Official Scheme? No
Geographic Reach National
Primary Audience Industry/Business
Results and Impact Invited presentation at Statistics Netherlands on attrition in surveys. Sparked interest in the problem of attrition in several longitudinal surveys that Statistics Netherlands is running. Am involved as consultant in a project that aims to limit future attrition for Statistics Netherlands
Year(s) Of Engagement Activity 2015