Enhancing and Enriching Historic Census Microdata
Lead Research Organisation:
University of Essex
Department Name: UK Data Archive
Abstract
This proposal is for the creation of Samples of Anonymised Records (SARs) from British censuses 1961 to 1981 and reconstruction of a 2001 SAR using record-level data recovered from archive tapes by the Office for National Statistics (ONS). Current changes to ONS IT infrastructure and some benefits from contemporary census processing present a unique moment of opportunity to recover these historical data and to create a rich new research resource for the analysis of social change over the last 50 years. The insights likely to be revealed from these data directly address ESRC's current priorities of 'Influencing behaviour and informing interventions' (e.g. 'How does the interplay of childhood, family, community and wider society influence inequalities in wellbeing?') and 'A vibrant and fair society' (e.g.,'How mobile is our society?'). In relation to both of these objectives ESRC has articulated a clear intention to make better use of existing data and to deepen the capacity of the UK research community.
Planned Impact
The main beneficiaries of these SARs will be the academic community, the impact of whose research activities is unpredictable.
| Description | The primary objectives of this project were to create a) preservation copies and b) researcher samples for the 1961, 1966, 1971, 1981 censuses. At the time of writing (13 November 2014) these objectives have not been entirely completed for various reasons which have been discussed with the ESRC Case Officer for the project. The preservation copies of the four censuses have been returned, fully documented to the data owners (Office for National Statistics, National Records of Scotland and Northern Ireland Statistics Research Agency). These are the fully cleaned and documented versions of the whole existing digital microdata for these censuses, and will allow these agencies to be able to create bespoke data products from them. This accounts for more than half of the effort employed on this project. The researcher samples for these censuses have been specified, and at the time of writing only one has passed statistical disclosure control. However, this one specification is the bench mark for the remainder. Samples will be made available at three levels of access: open, safeguarded and controlled, and depending on year, at both individual level and household levels. The first element of this project has effectively ensured the continued survival of usable microdata files from these censuses; the second element will put researcher quality datasets in front of different types of users and researchers. These will allow a long-term analysis of changes at this microlevel in society and economy for the first time. A large range of new research opportunities will be opened up by this project. Possibilities include examination of (im)migrants in 1971 with richer data; tracking deindustrialisation and the feminisation of the labour force and its relationship to the family, and analysis of quality of life within specific geographic areas to inform health studies, especially amongst the development of the new towns in the 1970s. We have discovered a large range of undocumented issues with the quality of underlying data - mostly issues relating to technology at the time. We note that all of the original census schedules still survive and that no information has been lost; however, some of the electronic records have not been fully recovered. One significant unintended consequence of this activity has been the explicit acceptance by data owners that as data 'ages' there are changes in the risk of disclosure. We have shown that mortality, alongside data quality, uncertainty through sampling, memory loss and churn can significantly reduce the disclosure risk. We believe that this evidence should allow NSIs and other data owners to make better provision for access to data through the long term without risking the disclosure of personal information. |
| Exploitation Route | The "findings" of this project are in many ways less important than the "products". However, the key finding is that it is imperative to properly document data collections at the time of creation, and that these data collections must be maintained over time, and refreshed and migrated as necessary. Our key outcome of getting our research on altered disclosure risk for older (census) data acknowledged by the NSIs can be used by others. |
| Sectors | Digital/Communication/Information Technologies (including Software),Government, Democracy and Justice |
| Title | 1961 Census Microdata Household File for Great Britain: 0.95% Sample |
| Description | The 1961 Census Microdata Household File for Great Britain: 0.95% Sample dataset was created from existing digital records from the 1961 Census |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8273&type=Data%20catalogue |
| Title | 1961 Census Microdata Individual File for Great Britain: 5% Sample |
| Description | The 1961 Census Microdata Individual File for Great Britain: 5% Sample dataset was created from existing digital records from the 1961 Census |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8272&type=Data%20catalogue |
| Title | 1961 Census Microdata Teaching Dataset for Great Britain: 1% Sample: Open Access |
| Description | The 1961 Census Microdata Teaching Dataset for Great Britain: 1% Sample: Open Access dataset was created from existing digital records from the 1961 Census. It can be used as a 'taster' file for 1961 Census data and is freely available for anyone to download under an Open Government Licence. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue?sn=8274#administrative |
| Title | 1961 Census Microdata for Great Britain: 9% Sample: Secure Access |
| Description | The 1961 Census Microdata for Great Britain: 9% Sample: Secure Access dataset was created from existing digital records from the 1961 Census. It comprises a larger population sample than the other files available from the 1961 Census and so contains sufficient information to constitute personal data, meaning that it is only available to Accredited Researchers, under restrictive Secure Access conditions. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8275&type=Data%20catalogue |
| Title | 1971 Census Microdata Household File for Great Britain: 0.95% Sample |
| Description | The 1971 Census Microdata Household File for Great Britain: 0.95% Sample dataset was created from existing digital records from the 1971 Census |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8269&type=Data%20catalogue |
| Title | 1971 Census Microdata Individual File for Great Britain: 5% Sample |
| Description | The 1971 Census Microdata Individual File for Great Britain: 5% Sample dataset was created from existing digital records from the 1971 Census |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8268&type=Data%20catalogue |
| Title | 1971 Census Microdata Teaching Dataset for Great Britain: 1% Sample: Open Access |
| Description | The 1971 Census Microdata Teaching Dataset for Great Britain: 1% Sample: Open Access dataset was created from existing digital records from the 1971 Census. It can be used as a 'taster' file for 1971 Census data and is freely available for anyone to download under an Open Government Licence. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue?sn=8270#administrative |
| Title | 1971 Census Microdata for Great Britain: 9% Sample: Secure Access |
| Description | The 1971 Census Microdata for Great Britain: 9% Sample: Secure Access dataset was created from existing digital records from the 1971 Census. It comprises a larger population sample than the other files available from the 1971 Census and so contains sufficient information to constitute personal data, meaning that it is only available to Accredited Researchers, under restrictive Secure Access conditions. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8271&type=Data%20catalogue |
| Title | 1981 Census Microdata Household File for Great Britain: 0.95% Sample |
| Description | The 1981 Census Microdata Household File for Great Britain: 0.95% Sample dataset was created from existing digital records from the 1981 Census |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8242&type=Data%20catalogue |
| Title | 1981 Census Microdata Individual File for Great Britain: 5% Sample |
| Description | The 1981 Census Microdata Individual File for Great Britain: 5% Sample dataset was created from existing digital records from the 1981 Census |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8241&type=Data%20catalogue |
| Title | 1981 Census Microdata Teaching Dataset for Great Britain:1% Sample: Open Access |
| Description | The 1981 Census Microdata Teaching Dataset for Great Britain: 1% Sample: Open Access dataset was created from existing digital records from the 1981 Census. It can be used as a 'taster' file for 1981 Census data and is freely available for anyone to download under an Open Government Licence. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue?sn=8243#administrative |
| Title | 1981 Census Microdata for Great Britain: 9% Sample: Secure Access |
| Description | The 1981 Census Microdata for Great Britain: 9% Sample: Secure Access dataset was created from existing digital records from the 1981 Census. It comprises a larger population sample than the other files available from the 1981 Census and so contains sufficient information to constitute personal data, meaning that it is only available to Accredited Researchers, under restrictive Secure Access conditions. |
| Type Of Material | Database/Collection of data |
| Year Produced | 2017 |
| Provided To Others? | Yes |
| Impact | Not known |
| URL | https://discover.ukdataservice.ac.uk/catalogue/?sn=8248&type=Data%20catalogue |