Hierarchical Analysis of Unit Nonresponse in Sample Surveys
Lead Research Organisation:
University of Southampton
Department Name: Statistical Sciences Research institute
Abstract
Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
Organisations
Publications
Chris Skinner (Author)
(2011)
Response propensity weighting for nonresponse using multilevel models
Durrant G
(2011)
Using Paradata to Predict Best Times of Contact, Conditioning on Household and Interviewer Influences
in Journal of the Royal Statistical Society Series A: Statistics in Society
Durrant G
(2010)
Effects of Interviewer Attitudes and Behaviors on Refusal in Household Surveys
in Public Opinion Quarterly
Durrant G
(2009)
Multilevel Modelling of Refusal and Non-Contact in Household Surveys: Evidence From Six UK Government Surveys
in Journal of the Royal Statistical Society Series A: Statistics in Society
Durrant G
(2009)
Imputation methods for handling item-nonresponse in practice: methodological issues and recent debates
in International Journal of Social Research Methodology
Durrant G
(2018)
Which Schools and Pupils Respond to Educational Achievement Surveys?: A Focus on the English Programme for International Student Assessment Sample
in Journal of the Royal Statistical Society Series A: Statistics in Society
Fiona Steele (Author)
(2009)
Alternative approaches to multilevel modelling of survey noncontact and refusal
Gabriele Durrant (Author)
(2009)
Using paradata to inform best times of contact : a multilevel event history modelling approach
Gabriele Durrant (Author)
(2011)
Interviewer effects on the process leading to cooperation or refusal using interviewer call record data
John Micklewright (Author)
(2010)
Modelling contextual non-response behaviour in a multistage survey of school children
| Description | The key objectives of this research grant were to explore the uses of paradata for cross-sectional and longitudinal surveys with the aim of gaining knowledge that leads to improvement in field process management and responsive survey designs. The development of methodology to make best use of such paradata in statistical models also played a key role. The research project was organised into three interlinked subprojects with the specific aims of: 1) investigating the use of call record data and interviewer observations to study nonresponse in cross-sectional and longitudinal surveys, 2) providing insights into the effects of interviewing strategies and attributes on response, and 3) gaining knowledge about the measurement error properties of paradata. The study has produced a wealth of findings which are outlined in detail in the 12 papers from this project. The key findings are described for each subproject: 1.) Paradata exhibits a complex multilevel structure which needs to be reflected in a statistical model analysing such data. Multilevel discrete time event history models were developed to analyse best times to establish contact and cooperation. Time varying call record information, such as features of the call history and of the current call, play a key role in predicting the outcome of each call. Interviewer observations, e.g. on the type and condition of the house and the presence of dependent children, proved to be useful for predicting the likelihood of cooperation. The novel application of sequence analysis of call record data identified a categorisation of sequences based on length and outcome. Subsequently the project developed models to predict length and final outcome of call sequences. We found that including information from the most recent call outcome very significantly improves prediction. The models developed are currently being used by Statistics Sweden to reduce the number of unproductive calls in the Swedish Labour Force Survey. When analysing the risk of nonresponse bias during data collection, the R-indicator, a measure commonly used after data collection, was shown not to perform well and the use of the delta index and the coefficient of variation are recommended instead. A key finding is that survey estimates tend to stabilise after about 5 calls indicating potential to stop calling after that reducing the number of unproductive calls made. 2.) The research developed multilevel cross-classified and multiple membership models to analyse interviewer effects in a longitudinal survey. The substantive results indicated strongly that the most recent interviewer has the highest influence of nonresponse across waves of a longitudinal survey contrasting earlier findings. The substantive findings confirm that interviewer experience, grade and continuity are significant predictors of non-response, whereas a significant role of interviewer personality traits could not be proven. Regarding the analysis of doorstep interactions, interviewer's experience and confidence of the interviewer were found to play a key role with more experienced and more confident interviewers showing higher likelihoods to achieve cooperation. 3.) Whilst paradata may be subject to measurement error our study revealed a generally high agreement between the interviewer observations and the Census reports, which implies that the interviewer observations we analyzed suffer from minimal measurement error. |
| Exploitation Route | The project reached out to a wide academic audience, comprising researchers in Statistics and Survey Methodology, and researchers in social science disciplines using data from social surveys. The impact of the research has been through academic publications and working papers (15 so far), an edited special issue in an international journal, more than 35 conference papers, 4 sessions at international conferences, an international research symposium and a short course. Furthermore, the research has already been taken forward by Statistical Offices, in particular Statistics Sweden, with which we are closely collaborating. They are currently implementing the models we developed on predicting length and outcome of call sequences to the Swedish Labour Force Survey. The aim is to reduce the number of unsuccessful call sequences leading to potentially very significant cost reductions. The UK Office for National Statistics has also expressed high interest in some of our findings and models, again with the aim of reducing the number of unproductive calls and to control for nonresponse bias. The work will also help market research organisations and other survey organisations to improve their data collection processes. The work on interviewer effects will impact on interviewer selection and training. Knowledge on the measurement error properties of paradata and interviewer observations will help to improve the collection of such data. The project also prepared the ground work for another successful grant, the Bayesian Adaptive Survey Design Network funded by the Leverhulme Trust. This project brings together international academic and non-academic experts in the area of paradata and survey designs to improve responsive survey designs, including the US Census Bureau, CBS Netherlands and the UK Office for National Statistics. Again, models and substantive findings from this project are being shared and developed further. The findings from the project also provided directly ideas for the successful ESRC NCRM research workpackage 1 grant. The work also informed directly the topics for 2 PhD theses. |
| Sectors | Government, Democracy and Justice |
| URL | http://www.southampton.ac.uk/s3ri/research/projects/the_use_of_paradata_in_cross_sectional_and_longitudinal_surveys.page#overview |
| Description | Findings on nonresponse analysis were in parts used by survey practitioners, e.g. by the Office for National Statistics. They re-considered the use and collection of paradata. They also used the findings from our research on the number of calls made to a household for reducing costs on their face-to-face surveys very significantly (number of calls reduced from 'without maximum' to 13 calls per household). |
| First Year Of Impact | 2017 |
| Sector | Government, Democracy and Justice |
| Impact Types | Societal,Economic,Policy & public services |
| Description | Non-response biases in surveys of school children : the case of the English PISA samples |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | 22nd International Workshop on Household Survey Nonresponse http://www.nonresponse.org/r/9/57/News/Nonresponse_Workshop_2011/ We analyse response patterns to an important survey of school children, exploiting rich auxiliary information on respondents' and non-respondents' cognitive ability that is correlated both with response and the learning achievement that the survey aims to measure. The survey is the Programme for International Student Assessment (PISA), which sets response thresh The audience was interested in the talk given. |
| Year(s) Of Engagement Activity | 2013 |
| Description | Using paradata to develop hierarchical response propensity models |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Geographic Reach | International |
| Primary Audience | Professional Practitioners |
| Results and Impact | Paper presented at the International Workshop on Household Nonresponse, in Slovenia. Establishing contact is an important part of the response process and effective interviewer calling patterns are critical in achieving contact and subsequent possible cooperation. Recent developments in the survey data collection process have led to the collection of so-called process data or paradata, which greatly extend the basic information on interviewer calls. The project to be presented aims to bui The paper received high interest from the audience. |
| Year(s) Of Engagement Activity | 2013 |