Qualitative Business Survey Data: An Assessment based on a Micro-Comparison with Quantitative Data
Lead Research Organisation:
National Institute of Economic and Social Research
Department Name: National Institute of Economic & Soc Res
Abstract
Abstracts are not currently available in GtR for all funded research. This is normally because the abstract was not required at the time of proposal submission, but may be because it included sensitive information such as personal details.
Publications
Holmes M
(2020)
Architects as Nowcasters of Housing Construction
in National Institute Economic Review
Lui S
(2011)
Qualitative Business Surveys: Signal or Noise?
in Journal of the Royal Statistical Society Series A: Statistics in Society
Lui S
(2011)
The utility of expectational data: Firm-level evidence using matched qualitative-quantitative UK surveys
in International Journal of Forecasting
Matheson T
(2009)
Nowcasting and predicting data revisions using panel survey data
in Journal of Forecasting
| Description | 1. Aims and Objectives This project assesses the relationship, at the firm-level, between qualitative business survey data produced by the Confederation of British Industry (CBI) and quantitative data collected by the Office for National Statistics (ONS). Interest focuses on the CBI survey because it appears more timely, although the cost of that timeliness is its qualitative nature. Firms are asked a range of questions to which they provide categorical instead of quantitative answers; e.g., they are asked whether output has fallen, stayed the same or risen. In addition to this retrospective question they are asked about their expectations of future output growth. Previous studies of the relationship between these qualitative and quantitative surveys have largely relied on comparisons at an aggregate level. By contrast, we compare the individual responses provided to the CBI with those collected by the ONS on a firm-by-firm basis. This project aims to improve economists' use of qualitative survey data by better understanding firms' responses to qualitative surveys. It ascertains the informational content of qualitative retrospective and prospective (expectational) survey data by examining the consistency between these data and the quantitative data provided by the same respondents to the ONS. It also seeks to deliver improved methods for nowcasting and forecasting economic activity using qualitative panel survey data. 2. Main Research Results The basis for this project was access, for the first time, to the firm-level data underlying both the CBI's Industrial Trends Survey (ITS) and the ONS's Monthly Production Inquiry (MPI). At the ONS's datalab, preserving confidentiality obligations, the two surveys were matched to relate a given firm's qualitative responses to the ITS with its quantitative responses to the MPI. This matched dataset was analysed to provide a definitive means of assessing the informational content of qualitative business surveys. Below we outline three main results. These, and the methods developed, constitute the principal academic achievements of this project. 2.1 Informational content of the CBI survey To assess formally the reliability of the CBI data we modelled the relationship between the ITS and MPI panel data, allowing for their dynamics, and tested various hypotheses about the former. These formalise whether the qualitative survey data are noise, or whether they contain a signal about the quantitative data. We find that the retrospective qualitative CBI data are plainly related to the responses the same firms gave to the MPI. Firms also appear to follow the CBI's instructions that they should report on output movements over the last three/four months 'quite' closely with the peak signal two relative to three months ago. However, on balance, the statistical evidence points to the signal remaining statistically significant up to about six months in the past; although given the MPI asks about sales while the ITS asks about output, this can arise even when firms do follow the CBI's instructions if unexpected fluctuations in sales growth are met from stocks. A clearer conclusion is that the firm-level qualitative data do not provide a good coincident indicator of growth. This was confirmed when, having introduced a novel means of inferring the official quantitative data from the qualitative data using Bayes' theorem, we found that conditioning autoregressive forecasts of the MPI data on contemporaneous values of the ITS data does not improve inference more than trivially. This suggests that the CBI survey has little role to play in enhancing our knowledge of what has recently happened to manufacturing output, a result which is confirmed when we examine the signal generated by macroeconomic data. 2.2 Utility of expectational data from the CBI survey To assesses the utility of qualitative expectational survey data at the firm-level in terms of both their ability to anticipate firms' subsequent retrospective, but qualitative, reports of their performance but also these same firms' quantitative answers we use nonparametric tests. These test the "best-case scenario" that firms have rational expectations and that reported expectations are 'optimal' predictions of future outcomes, in the sense that they minimise the firm's loss function. We introduce a test for the coherence between the two surveys. This tests whether the two samples are measuring the same concept of output growth, as implicitly assumed when qualitative surveys are used to infer quantitative data. While firms' qualitative expectations of their future output growth are best-case predictions of their retrospective qualitative assessment of this growth, they do not contain a signal about the quantitative data. But the retrospective qualitative data are coherent with the quantitative data, suggesting that the two surveys are measuring similar concepts despite different questions and different sampling assumptions. The apparent paradox that the qualitative expectational data do not help predict the quantitative realisations data might be explained by 'forecasting' and 'discretisation' errors confounding any signal from the expectational data. Moreover, when reporting their expectations qualitatively firms may report the mode or median of their subjective density forecast rather than its mean. Therefore, qualitative business survey data, given that they are published ahead of the ONS's quantitative data, are likely to prove more useful for nowcasting than forecasting. It would help if the surveys elicited quantitative rather than qualitative responses. Information is lost when firms discretise their responses. 2.3 Collective Sentiment Firms, when replying to qualitative surveys, react not just to the hard (quantitative) facts but to the "herd". Sentiment, in other words, has a common collective component, as well as individual components. This feedback explains the tendency for the balance statistic to under/over react to macroeconomic events. |
| Exploitation Route | A range of important economic issues can be better understood through increased access and exploitation of disaggregate qualitative and quantitative panel data. Firstly, by matching the MPI to the monthly stocks inquiry, at the firm-level, we plan to study the relationship between stock movements and turnover. This exercise could also be undertaken at the quarterly level to explore stock holding in retailing. Thereby, we can examine the role stock movements have played in the dynamics of the current recession. Anecdotally they have been playing an important role and access to these disaggregate data would enable empirical testing. Secondly, and more generally, using these disaggregate panel data we can examine the relationship between the (aggregate) business cycle and cyclical movements at the firm-level. Thirdly, by matching firm-level data from the ONS, not just for manufacturing but for the service sector also, with qualitative business survey data from the CBI we can look at questions like whether firms which report credit reports (identified in the CBI survey, given qualitative surveys ask a range of questions not asked in official surveys) cut output. Fourthly, we should like to consider whether improved nowcasts and forecasts can be obtained by modelling these disaggregate data. Use of these large disaggregate panel data sets, both qualitative and quantitative, would also involve the development of econometric tools. This would include factor models that handle both qualitative and quantitative panel data. |
| Sectors | Financial Services, and Management Consultancy,Government, Democracy and Justice |
| Description | Business surveys : signal or noise? |
| Form Of Engagement Activity | Participation in an activity, workshop or similar |
| Part Of Official Scheme? | No |
| Primary Audience | |
| Results and Impact | Seminar given at the ESRC Festival of Social Science 2008, hosted by the Confederation of British Industry (CBI). |
| Year(s) Of Engagement Activity | |
| URL | http://www.niesr.ac.uk/event/ESRC%20Festival%20of%20Social%20Science%20-%20Business%20surveys%20sign... |
| Description | Business surveys : signal or noise? |
| Form Of Engagement Activity | A talk or presentation |
| Part Of Official Scheme? | No |
| Primary Audience | |
| Results and Impact | Seminar organised as part of the ESRC Festival of Social Science 2008. Confederation of British Industry, London (10 March 2008) |
| Year(s) Of Engagement Activity | 2008 |
| URL | http://www.niesr.ac.uk/event/ESRC%20Festival%20of%20Social%20Science%20-%20Business%20surveys%20sign... |