Survey data quality in panel surveys: trade-offs between nonresponse and measurement errors

This project aims to improve data quality in panel surveys, while at the same time lowering survey costs. Panel surveys involve groups of people or households that are followed over time. Typically, they answer questions annually, about their life, attitudes and behaviours. From the resulting data from these surveys, we can learn a lot about life in Britain. This is especially the case when a survey documents the life of a representative sample of people, as is the case in the British Household Panel Study (BHPS). From the BHPS we can learn how many people are unemployed every year, who they are, and how and when they find a job again for example. A return to employment is in turn likely to affect household income, and family wellbeing. In short, panel studies have the power to enrich our knowledge about how people's lives change over time, at the individual level, and in Britain as a whole.

This only is the case however when the measurement of all topics of interest are without error. There are two sources of error that threaten to make panel data invalid and unreliable. First, nonresponse among specific respondents, and second errors in measurement of the topic of interest using survey questions. Nonresponse is particularly likely to lead to biased data when specific groups of respondents are likely to drop out. This might be exactly the case for people who for example move because they have found a new job. Earlier research has suggested that people are less likely to participate in a survey when they are in the process of moving house. The reason for moving is however often a change of job. This in turn leads the survey data to under represent people who change jobs; one of the topics of interest in panel surveys.

Similarly, measurement error might also occur for specific groups of people. For example, people with lower cognitive abilities have more difficulty understanding questions, and remembering any information required to give a correct answer. This in turn might lead to biased data when people are asked about their knowledge and opinions of policy issues.

Survey methodologists worry that errors due to nonresponse and measurement interact. Some reasons for nonresponse might at the same time also be a reason for reporting with more measurement error. Lower cognitive abilities, complex income compositions, or language difficulties are among the reasons for both nonresponse and measurement error.
In this proposal, trade-offs and common causes for both nonresponse error and measurement error are studied.

The first part of this project is devoted to developing models to separate the two types of error; we normally have no information on measurement error for those people that do not participate, so statistical models are necessary to overcome this problem. The second part studies common causes of the two error sources, and identifies which respondent characteristics are responsible for them. The final part studies the trade-offs between measurement error and nonresponse. Understanding the trade-off better will enable researchers to compare the nature and size of both errors, and make better informed decisions in trying to limit survey errors. This in turn will make future panel studies more cost-effective and provide higher data quality.

Longitudinal survey data are the source for many social scientists, economists, and policy makers who study or take policy-decisions regarding poverty, income and well-being. In the UK, research based on longitudinal surveys such as the British Household Panel Survey (BHPS), the Birth Cohort Studies and the English Longitudinal Study of Aging (ELSA) has over many years had a huge impact on policy debates and policy decisions. This study should improve the quality of such data and of the research based upon those data.

Specifically, any lessons regarding improved analysis approaches that better take into account measurement error, non-response error and the interaction between them will be of use to analysts across the broad range of disciplines that use longitudinal survey data, including sociology, economics, geography, psychology, politics, health sciences and demography. Although this includes research in academia, it also includes researchers in market-research and government agencies; for example the home office, and Office for National Statistics.

Market researchers and survey agencies perform much of the fieldwork for companies, government agencies and academia. Market researchers have over the years faced increasing pressure on the quality and costs of doing surveys. Because of increasing nonresponse rates and the higher survey costs associated with it, many market researchers have left traditional sampling methods, and have instead opted for using non-probability samples in the form of volunteer opt-in panels. Even within those panels however, nonresponse among panellists is a large problem. Market researchers are also worried about the answering behaviour of respondents in their panel surveys. Respondents often do not take the time and effort to complete surveys seriously, and instead opt for a strategy that takes the least effort against the least amount of time, resulting in strategies as "nondifferentiation", " straightlining" and "speeding".
Such undesirable answer strategies are often dealt with using 'ad hoc' methods. The guidelines that this project will produce will allow market researchers to easily study the extent of nonresponse and measurement error in their panel studies. The guidelines will enable market researchers to decide whether to convince unactive members to become active again, and whether measurement error really is as large as a problem as many suspect it is. Finally, market researchers will be able to use the guidelines to whether nonresponse or measurement error is the largest problem in their survey. The statistical models that this project will use are rather complicate for some people without a solid statistical background, The guidelines will be practical however so that everyone can use and implement them. If any statistical models are necessary, specific programming syntax for all major statistical packages will be added to the guideline as an appendix. To this end, we will actively work together with the Market Research Society (MRS) in designing and writing down these guidelines. We will also collaborate with the MRS to disseminate the guidelines to market researchers and survey agencies.

Government agencies will benefit from this project in two ways. First, by the improved data quality of the panel surveys it uses for decision making, leading to better informed policy-decisions. Second, by the lower costs of panel surveys, meaning that funding costs of future panel surveys will be lower.

It should be noted that academic researchers will also benefit from improvements that are implemented by the non-academic sectors, notably government and private sector survey agencies. Academics use a lot of data collected outside of academia so the impacts on non-academic audiences (see "Pathways to Impact") are highly relevant for academics too.