Survey data quality in panel surveys: trade-offs between nonresponse and measurement errors
Lead Research Organisation:
University of Essex
Department Name: Inst for Social and Economic Research
Abstract
This project aims to improve data quality in panel surveys, while at the same time lowering survey costs. Panel surveys involve groups of people or households that are followed over time. Typically, they answer questions annually, about their life, attitudes and behaviours. From the resulting data from these surveys, we can learn a lot about life in Britain. This is especially the case when a survey documents the life of a representative sample of people, as is the case in the British Household Panel Study (BHPS). From the BHPS we can learn how many people are unemployed every year, who they are, and how and when they find a job again for example. A return to employment is in turn likely to affect household income, and family wellbeing. In short, panel studies have the power to enrich our knowledge about how people's lives change over time, at the individual level, and in Britain as a whole.
This only is the case however when the measurement of all topics of interest are without error. There are two sources of error that threaten to make panel data invalid and unreliable. First, nonresponse among specific respondents, and second errors in measurement of the topic of interest using survey questions. Nonresponse is particularly likely to lead to biased data when specific groups of respondents are likely to drop out. This might be exactly the case for people who for example move because they have found a new job. Earlier research has suggested that people are less likely to participate in a survey when they are in the process of moving house. The reason for moving is however often a change of job. This in turn leads the survey data to under represent people who change jobs; one of the topics of interest in panel surveys.
Similarly, measurement error might also occur for specific groups of people. For example, people with lower cognitive abilities have more difficulty understanding questions, and remembering any information required to give a correct answer. This in turn might lead to biased data when people are asked about their knowledge and opinions of policy issues.
Survey methodologists worry that errors due to nonresponse and measurement interact. Some reasons for nonresponse might at the same time also be a reason for reporting with more measurement error. Lower cognitive abilities, complex income compositions, or language difficulties are among the reasons for both nonresponse and measurement error.
In this proposal, trade-offs and common causes for both nonresponse error and measurement error are studied.
The first part of this project is devoted to developing models to separate the two types of error; we normally have no information on measurement error for those people that do not participate, so statistical models are necessary to overcome this problem. The second part studies common causes of the two error sources, and identifies which respondent characteristics are responsible for them. The final part studies the trade-offs between measurement error and nonresponse. Understanding the trade-off better will enable researchers to compare the nature and size of both errors, and make better informed decisions in trying to limit survey errors. This in turn will make future panel studies more cost-effective and provide higher data quality.
This only is the case however when the measurement of all topics of interest are without error. There are two sources of error that threaten to make panel data invalid and unreliable. First, nonresponse among specific respondents, and second errors in measurement of the topic of interest using survey questions. Nonresponse is particularly likely to lead to biased data when specific groups of respondents are likely to drop out. This might be exactly the case for people who for example move because they have found a new job. Earlier research has suggested that people are less likely to participate in a survey when they are in the process of moving house. The reason for moving is however often a change of job. This in turn leads the survey data to under represent people who change jobs; one of the topics of interest in panel surveys.
Similarly, measurement error might also occur for specific groups of people. For example, people with lower cognitive abilities have more difficulty understanding questions, and remembering any information required to give a correct answer. This in turn might lead to biased data when people are asked about their knowledge and opinions of policy issues.
Survey methodologists worry that errors due to nonresponse and measurement interact. Some reasons for nonresponse might at the same time also be a reason for reporting with more measurement error. Lower cognitive abilities, complex income compositions, or language difficulties are among the reasons for both nonresponse and measurement error.
In this proposal, trade-offs and common causes for both nonresponse error and measurement error are studied.
The first part of this project is devoted to developing models to separate the two types of error; we normally have no information on measurement error for those people that do not participate, so statistical models are necessary to overcome this problem. The second part studies common causes of the two error sources, and identifies which respondent characteristics are responsible for them. The final part studies the trade-offs between measurement error and nonresponse. Understanding the trade-off better will enable researchers to compare the nature and size of both errors, and make better informed decisions in trying to limit survey errors. This in turn will make future panel studies more cost-effective and provide higher data quality.
Planned Impact
Longitudinal survey data are the source for many social scientists, economists, and policy makers who study or take policy-decisions regarding poverty, income and well-being. In the UK, research based on longitudinal surveys such as the British Household Panel Survey (BHPS), the Birth Cohort Studies and the English Longitudinal Study of Aging (ELSA) has over many years had a huge impact on policy debates and policy decisions. This study should improve the quality of such data and of the research based upon those data.
Specifically, any lessons regarding improved analysis approaches that better take into account measurement error, non-response error and the interaction between them will be of use to analysts across the broad range of disciplines that use longitudinal survey data, including sociology, economics, geography, psychology, politics, health sciences and demography. Although this includes research in academia, it also includes researchers in market-research and government agencies; for example the home office, and Office for National Statistics.
Market researchers and survey agencies perform much of the fieldwork for companies, government agencies and academia. Market researchers have over the years faced increasing pressure on the quality and costs of doing surveys. Because of increasing nonresponse rates and the higher survey costs associated with it, many market researchers have left traditional sampling methods, and have instead opted for using non-probability samples in the form of volunteer opt-in panels. Even within those panels however, nonresponse among panellists is a large problem. Market researchers are also worried about the answering behaviour of respondents in their panel surveys. Respondents often do not take the time and effort to complete surveys seriously, and instead opt for a strategy that takes the least effort against the least amount of time, resulting in strategies as "nondifferentiation", " straightlining" and "speeding".
Such undesirable answer strategies are often dealt with using 'ad hoc' methods. The guidelines that this project will produce will allow market researchers to easily study the extent of nonresponse and measurement error in their panel studies. The guidelines will enable market researchers to decide whether to convince unactive members to become active again, and whether measurement error really is as large as a problem as many suspect it is. Finally, market researchers will be able to use the guidelines to whether nonresponse or measurement error is the largest problem in their survey. The statistical models that this project will use are rather complicate for some people without a solid statistical background, The guidelines will be practical however so that everyone can use and implement them. If any statistical models are necessary, specific programming syntax for all major statistical packages will be added to the guideline as an appendix. To this end, we will actively work together with the Market Research Society (MRS) in designing and writing down these guidelines. We will also collaborate with the MRS to disseminate the guidelines to market researchers and survey agencies.
Government agencies will benefit from this project in two ways. First, by the improved data quality of the panel surveys it uses for decision making, leading to better informed policy-decisions. Second, by the lower costs of panel surveys, meaning that funding costs of future panel surveys will be lower.
It should be noted that academic researchers will also benefit from improvements that are implemented by the non-academic sectors, notably government and private sector survey agencies. Academics use a lot of data collected outside of academia so the impacts on non-academic audiences (see "Pathways to Impact") are highly relevant for academics too.
Specifically, any lessons regarding improved analysis approaches that better take into account measurement error, non-response error and the interaction between them will be of use to analysts across the broad range of disciplines that use longitudinal survey data, including sociology, economics, geography, psychology, politics, health sciences and demography. Although this includes research in academia, it also includes researchers in market-research and government agencies; for example the home office, and Office for National Statistics.
Market researchers and survey agencies perform much of the fieldwork for companies, government agencies and academia. Market researchers have over the years faced increasing pressure on the quality and costs of doing surveys. Because of increasing nonresponse rates and the higher survey costs associated with it, many market researchers have left traditional sampling methods, and have instead opted for using non-probability samples in the form of volunteer opt-in panels. Even within those panels however, nonresponse among panellists is a large problem. Market researchers are also worried about the answering behaviour of respondents in their panel surveys. Respondents often do not take the time and effort to complete surveys seriously, and instead opt for a strategy that takes the least effort against the least amount of time, resulting in strategies as "nondifferentiation", " straightlining" and "speeding".
Such undesirable answer strategies are often dealt with using 'ad hoc' methods. The guidelines that this project will produce will allow market researchers to easily study the extent of nonresponse and measurement error in their panel studies. The guidelines will enable market researchers to decide whether to convince unactive members to become active again, and whether measurement error really is as large as a problem as many suspect it is. Finally, market researchers will be able to use the guidelines to whether nonresponse or measurement error is the largest problem in their survey. The statistical models that this project will use are rather complicate for some people without a solid statistical background, The guidelines will be practical however so that everyone can use and implement them. If any statistical models are necessary, specific programming syntax for all major statistical packages will be added to the guideline as an appendix. To this end, we will actively work together with the Market Research Society (MRS) in designing and writing down these guidelines. We will also collaborate with the MRS to disseminate the guidelines to market researchers and survey agencies.
Government agencies will benefit from this project in two ways. First, by the improved data quality of the panel surveys it uses for decision making, leading to better informed policy-decisions. Second, by the lower costs of panel surveys, meaning that funding costs of future panel surveys will be lower.
It should be noted that academic researchers will also benefit from improvements that are implemented by the non-academic sectors, notably government and private sector survey agencies. Academics use a lot of data collected outside of academia so the impacts on non-academic audiences (see "Pathways to Impact") are highly relevant for academics too.
People |
ORCID iD |
Peter Lugtig (Principal Investigator) |
Publications
De Leeuw E
(2014)
Wiley StatsRef: Statistics Reference Online
Lynn P
(2017)
Total Survey Error in Practice
Lugtig P
(2015)
The Use of PCs, Smartphones, and Tablets in a Probability-Based Panel Survey Effects on Survey Measurement Error
in Social Science Computer Review
Lugtig P.
(2017)
The relative size of measurement error and attrition error in a panel survey. Comparing them with a new multi-trait multi-method model
in Survey Research Methods
Lugtig, P.
(2017)
The relative size of measurement error and attrition error in a panel survey. Comparing them using a new Multi-Trait Multi-Method model.
in Survey Research Methods
Lugtig P
(2014)
Panel Attrition Separating Stayers, Fast Attriters, Gradual Attriters, and Lurkers
in Sociological Methods & Research
Description | - Only specific forms of attrition in a survey can be predicted. Most notably attrition due to refusal by the respondent. - Respondents who drop out from surveys have a different profile from respondents who continue to participate in panel surveys. The most notable differences between the two types are not merely demographic, but also psychographic. More conscientious and introvert people are more likely to become 'loyal' panel members. - Item missings, and lower cooperation rates by the interviewer are good predictors of later dropout from a panel study. - A theoretical model for understanding trade-offs between nonresponse and measurement error has been developed and described. - A statistical model to estimate the size of both nonresponse and measurement error has been developed. |
Exploitation Route | Findings can be used to inform decision-making in so-called adaptive survey designs. These design gain a lot of interest in recent years from survey methodologists and survey practioners. The project has found for example that specific forms of attrition (due to noncontact, refusal etc.) can be predicted to a different extent. Refusals are the one cause for attrition that can be predicted fairly well. Using covariates, I have identified who will attrite. This finding can be used to target those likely refusers, and try to prevent them attriting, or perhaps to re-recruit them into the survey. |
Sectors | Government, Democracy and Justice,Other |
URL | http://www.peterlugtig.com |
Description | Findings from my research have been used by survey practitioners. Specifically the statistical offices in the Netherlands and Sweden, to inform the design of (future) longitudinal studies. |
First Year Of Impact | 2013 |
Sector | Government, Democracy and Justice,Other |
Impact Types | Policy & public services |
Description | Statistics Netherlands: advisory and collaborative work |
Geographic Reach | National |
Policy Influence Type | Participation in a guidance/advisory committee |
Impact | Participated in consultation of Statistics Netherlands on attrition in several longitudinal surveys they are running. As of 2016, we are setting up a formal research network to facilitate consultation on survey methodology, and collaborative research on the prevention of attrition |
Description | WIn project |
Organisation | Statistics Netherlands |
Country | Netherlands |
Sector | Public |
PI Contribution | Set up a collaborating rpoject with statistics netherlands investigating the use of smartphone apps as a way to build an ongoing longtitudinal survey in the Netherlands. Together we have got funding to build an app (2 FtE), test it (1 FTE), and make the data usable for official statistics (1 FTE) |
Collaborator Contribution | see above |
Impact | Bais, F., Schouten, J.G., Lugtig, P., Toepoel, V., Arends-Toth, J., Douhou, S., Kieruj, N., Morren, M. and Vis, C. (2017) Can survey item characteristics relevant to mode-specific measurement error be coded reliably? Sociological Methods & Research Lugtig, P, Toepoel, V., Haan, M. and Schouten, J.G. (2017) Het Waarneem Innovatie Netwerk. Stator, 3, 16-19 |
Start Year | 2016 |
Description | 2 presentations at TSE conference 2015 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | One invited presentation on ' Total survey error for longitudinal surveys' (with peter Lynn) One contributed presentation on 'a common metric for nonresponse and measurement error'. Sparked discussion at conference. Materials of both chapters are now used in course at University of Michigan on total survey error |
Year(s) Of Engagement Activity | 2015 |
Description | Blog post about item-missings and relation to attrition |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other academic audiences (collaborators, peers etc.) |
Results and Impact | had requests for paper that will be based on blog post not aware of any impact |
Year(s) Of Engagement Activity | 2014 |
URL | http://www.peterlugtig.com/2014/04/are-item-missings-related-to-later.html |
Description | ESRA conference, Reykjavik |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Gave three presentations at ESRA conference. This is the largest conference for survey researchers in Europe. Based on presentations Ireceived some feedback on presentations which helped in improving papers. Two future collaborations with industry in the field of attrition on smartphone panel surveys resulted from this. |
Year(s) Of Engagement Activity | 2015 |
Description | Invited talk at GESIS summerschool about survey errors |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | Invited lecture on importance of understanding survey errors to participants of summer school in survey methods led to a few students choosing to do a specific methods-related part in their ph.d. thesis |
Year(s) Of Engagement Activity | 2013 |
Description | Nonresponse workshop |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Discussion meeting of about 50 people from academia and national statistical offices, with the goal to exchange information on survey practice and theory. Led to invited talks at institutes, and potential collaboration with Statistics Sweden collaboration with statistical office of Sweden on designing longitudinal surveys |
Year(s) Of Engagement Activity | 2013,2014 |
URL | http://www.nonresponse.org |
Description | blog post: do respondents become sloppy before attrition |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other academic audiences (collaborators, peers etc.) |
Results and Impact | had discussion on forum about different ways to model attrition not aware of impact |
Year(s) Of Engagement Activity | 2014 |
URL | http://www.peterlugtig.com/2014/03/do-respondents-become-sloppy-before.html |
Description | blogpost: adaptive panel designs |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | blogpost describing how the findings from ESRC project on the trade-off between measurement and nonresponse error can be used to inform adaptive designs, where data collection protocols are adapted to the inidividual and fieldwork data. |
Year(s) Of Engagement Activity | 2015 |
URL | http://www.peterlugtig.com/2015/05/adaptive-designs-4-ways-to-improve.html |
Description | blogpost: longitudinal interview outcome data reduction |
Form Of Engagement Activity | A magazine, newsletter or online publication |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other academic audiences (collaborators, peers etc.) |
Results and Impact | had discussion at weblog about using Latent Classes or sequence analysis to model longitdinal interview outcome data. not aware |
Year(s) Of Engagement Activity | 2014 |
Description | blogpost: the tarditional web survey is dead |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | Blogpost on how web surveys are changing due to increased use of smartphones and tablets. Reached about 2.000 people through twitter, 500 people on blog directly. Received 10 re-tweets. |
Year(s) Of Engagement Activity | 2016 |
URL | http://www.peterlugtig.com/2016/02/the-old-web-survey-is-dead.html |
Description | blogpost: why panel surveys need to go adaptive |
Form Of Engagement Activity | Engagement focused website, blog or social media channel |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Postgraduate students |
Results and Impact | Blogpost in which I describe the increasing extent of attrition in longitudinal surveys in Europe, and what should be done to prevent this. Blogpost generated debate on weblog, and subsequent conferences. |
Year(s) Of Engagement Activity | 2015 |
URL | http://www.peterlugtig.com/2015/01/why-panel-surveys-need-to-go-adaptive.html |
Description | invited talk at Institute of Education, London on January 21, 2014 |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | Local |
Primary Audience | Other academic audiences (collaborators, peers etc.) |
Results and Impact | 50 people from Institute of Education attended my talk on attrition from longitudinal surveys Led to future collaboration between me and Institute of Education |
Year(s) Of Engagement Activity | 2014 |
Description | participation at AAPOR conference |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Professional Practitioners |
Results and Impact | Talks at 2013 and 2014 conferences of the American Association for Public Opinion Research about ESRC projects and research that was part of project. Had 1 talk in 2013, and 3 talks at conference, which is most high-profile conference in the field of survey methods Had requests for paper, and collaborations. |
Year(s) Of Engagement Activity | 2013,2014 |
URL | http://www.aapor.org |
Description | participation in Panel Survey Methods Workshop |
Form Of Engagement Activity | Participation in an activity, workshop or similar |
Part Of Official Scheme? | No |
Geographic Reach | International |
Primary Audience | Other academic audiences (collaborators, peers etc.) |
Results and Impact | 50 people attended my talk and talks of others at conference, which spearked discussion about design of longitudinal surveys No notable impacts, apart from information sharing and discussion. Led to ideas for related future research |
Year(s) Of Engagement Activity | 2014 |
URL | http://www.panelsurveymethods.wordpress.com |
Description | presentation at GESIS/Mannheim seminar series |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Professional Practitioners |
Results and Impact | Gave a presentation on ESRC project findings in a joint seminar series on ' research methods' of GESIS and the University of Mannheim, Germany. 1 april 2015 |
Year(s) Of Engagement Activity | 2015 |
Description | talk at Statistics Netherlands |
Form Of Engagement Activity | A talk or presentation |
Part Of Official Scheme? | No |
Geographic Reach | National |
Primary Audience | Industry/Business |
Results and Impact | Invited presentation at Statistics Netherlands on attrition in surveys. Sparked interest in the problem of attrition in several longitudinal surveys that Statistics Netherlands is running. Am involved as consultant in a project that aims to limit future attrition for Statistics Netherlands |
Year(s) Of Engagement Activity | 2015 |