Traces though Time: Prosopography in practice across Big Data

The growing availability of large digital historical datasets, coupled with the emergence of new methodologies and computer algorithms has the potential to revolutionise research in the Arts and Humanities. The right Big Data tools and approaches will deliver the potential to conduct research on the scale of entire populations - addressing key research questions and offering new insights.

Significant investment has already been sunk into the creation of large-scale digital resources. This investment is delivering historical big data of a variety, complexity and coverage that is beyond the scope of existing analytical tools and techniques. Yet these tools have not yet been the subject of large investment. Researchers in this field now require rapid innovation to extend the Big Data approaches pioneered for scientific and business applications, adapting and refining these to deliver practical analytical tools to support large-scale exploration of big historical datasets.

This innovative, multi-disciplinary project will address this challenge, bringing together international research experience in the digital humanities, natural language processing, information science, data mining and linked data, with large, complex and diverse 'big data' spanning over 500 years of British history.

The project's technical outputs will be a methodology and supporting toolkit that identify individuals within and across historical datasets, allowing people to be traced through the records and enabling their stories to emerge from the data. The tools will handle the 'fuzzy' nature of historical data, including aliases, incomplete information, spelling variations and the errors that are inevitably encountered in official records. The toolkit will be open and configurable, offering the flexibility to formulate and ask interesting questions of the data, exploring it in ways that were not imagined when the records were created. The open approach will create opportunities for further enhancement or re-use and offers the further potential to deliver the outputs as a service, extensible to new datasets as these become available. This brings the vision of 'bring your own data' closer, to find and link individuals in new combinations of datasets, from the widest range of historical sources.

The project will benefit academic and leisure historians alike, across the whole spectrum of digital history:
* It will assist historians seeking evidence of life-events through a collective study of individual biographies.
* It will help genealogists find and trace the paths of their ancestors across the landscape of the official record.
* It will help researchers by signposting routes between historical collections, enabling links between datasets at a deep level and creating opportunities for discovery.
* For cultural organizations it will illuminate effective approaches to creation and curation of new digital datasets to optimise their potential for linking and re-use.
* It will provide evidence to support policy making, helping balance the demands of Data Protection and information assurance with those of open data and Freedom of Information.
* It will provide a methodology to underpin the creation of new tools and resources, supporting the digital economy.

The project aims to extend the boundaries of current research in three important directions: to increase the extent and diversity of the data that can be handled; to improve support for inconsistent or fuzzy data; and to enable confidence measures to be tailored to fit specific research aims. These advances will extend the practical application of data linking techniques, enabling them to be applied to the large, diverse datasets that are continually emerging, to help answer historical research questions at a macro and micro scale. Our vision is to create a generic, extensible approach to tracing the lives of real people: through time and across the documentary evidence that survives them.

The 2012 Heritage Counts survey highlights that successful heritage organisations are 'anchor institutions' - the civic, cultural and intellectual institutions that help make places resilient. Creating routes into Big Data and equipping users to engage with our shared documentary heritage is vital and supports that resilience into a new digital age. We propose differing but complementary strategies, to maximise impact.

The 2012 ARA/CIPFA Public Services Quality Group report states that 56% of visitors to archives described the purpose of their visit as 'family history research'. Developing new tools for navigating Big Data will equip family and leisure historians to find and trace the paths of individuals across the landscape of the official record and beyond. Regular talks and events will highlight findings, users will benefit from new exhibitions showcasing the stories that emerge from the data and updated research guidance will support them in applying the tools in pursuit of their own investigations.
TNA's User Advisory Group will be consulted to surface user needs. Articles submitted to popular history magazines will further engage genealogists and local history organisations. The research team will post updates and podcasts and blog on popular themes such as the First World War, where we anticipate a wealth of stories emerging from the records. We will hold a 'hack day' for researchers to work on their own datasets alongside our developers. The Institute of Historical Research will host the final dissemination event, for scholars, user groups and educators across disciplines. An advisory board will be established, drawing on expertise from previous work, such as the AHRC funded Fine Rolls and ESRC funded Integrated census microdata project.
TNA's press office, licensing and commercial partners are used to handling significant attention around file releases and public interest stories. All partners will draw on their wide network of international media, publishing and business contacts to best promote the project and its findings.

As a government department and an executive agency of the Ministry of Justice, TNA is well placed to facilitate connections with policy makers. This research has the potential to impact policy in the Open Government Data, Transparency, Data Protection and privacy arenas by offering a robust approach to linking individuals across datasets, and providing evidence to underpin policy development.

For archives and cultural heritage organisations, this work will inform future digitisation, data creation and capture methodologies and will encourage archive services and other data to work together to enable exploration across their holdings. For public sector data holders, it will enable both greater access to records that need not be closed, and greater assurance that records are not being opened inappropriately. This latter has enormous implications for the support of information security and assurance.

TNA has close ties with UK public sector bodies and cultural organisations. We work closely with the Imperial War Museum and British Library to share knowledge. These relationships will help extend the methodology to ensure application to other collections. The availability of a robust methodology for linking personal entities has the potential to dramatically change retention, preservation and cataloguing practices.

The research team will use their links to schools (TNA's award winning Education & Outreach team) to work with pupils and teachers to delve into the stories behind the history and to engage the smartphone generation. Though our case studies, students will learn to engage with Big Data, both from a historical perspective and as it impacts the lives of young people growing up in a digital age. This will contribute to government's aspiration to develop digital knowledge and skills for the future, which, in turn supports UKplc to become more agile and innovative.