Big UK Domain Data for the Arts and Humanities (BUDDAH)

Web archives are an increasingly important resource for arts and humanities researchers, yet we have neither the expertise nor the tools to use them effectively. Both the data itself and the process of collection are poorly understood, and it is possible only to draw the broadest of conclusions from current analytical analysis. The proposed project will focus on deeper analysis of the dataset derived from the UK domain crawl from 1996 to 2013 (that is, when legal deposit legislation was extended to cover digital materials), totalling approximately 65 terabytes and constituting many billions of words. For the arts and humanities, this is very big data indeed.

A key objective of the project will be to develop a theoretical and methodological framework within which to study this data, which will be applicable to the much larger on-going UK domain crawl, as well as in other national contexts. Researchers will work with developers at the British Library to co-produce tools which will support their requirements, testing different methods and approaches. These may include, but are not limited to, sentiment analysis, topic modelling, proximity searching, link analysis, and geo-spatial analysis. The scale and inter-connectedness of the data requires an analytical, big data approach rather than the rendering of individual web pages.

A major study of the history of UK web space from 1996 to 2013, including language, file formats, the development of multimedia content, shifts in power and access, and so on, will be complemented by a series of sub-projects from a range of disciplines, for example contemporary history, literature, gender studies and material culture.

Project outputs will include a suite of tools associated with the 1996-2013 dataset; a series of case studies produced by the sub-projects; an online training course for arts and humanities researchers; peer-reviewed journal articles; and a monograph on the history of the UK web during this period.

The proposed research will have a significant impact on a range of audiences and sectors, including:

1. Institutions with responsibility for web archiving and digital preservation
The UK web domain dataset 1996-2013 is held by the British Library, which as of April 2013 also has responsibility for preserving data arising from a regular crawl of UK web space. This is a major extension of the BL's legal deposit role, and one which will require considerable investment in time and resource. If this work is to be of genuine value to researchers in the arts and humanities, it is vital to understand how they wish to use the web archive, both now and in the future, and the methods and tools that they will require to interrogate and represent the data effectively. This project will inform the BL's future strategy, by engaging arts and humanities researchers in the co-production of tools for analytical access to the data. It will also help curators to understand the data for which they are now responsible, and to refine their processes of collection. Arts and humanities researchers in turn will benefit from the expertise of the digital scholarship and web archive teams at the BL, and the project will also inform provision at the UK's other legal deposit libraries.

The National Archives (TNA) has responsibility for preserving UK government information published on the web, a more selective process than the wider UK domain crawl. Members of TNA staff will be invited to attend the two proposed workshops and to join the project advisory board. Arrangements for the harvesting of national web data, and provision of access to it, vary widely between countries, but many of the problems are common. Consequently, the findings of this project will feed in to international web archive and digital preservation practice internationally, and particularly within the EU.

2. Memory institutions
As the web becomes an increasingly important primary source, museums, galleries and other memory institutions will face the challenge of incorporating material derived from web archives in exhibitions and other forms of public engagement (e.g. the recent British Library Propaganda exhibition made use of archived digital materials). A more accessible and well-promoted archive, with an associated body of expertise, will support the imaginative use of big web data in this context. One of the project case studies will examine material culture and approaches to exhibiting the web.

3. Government, policy-makers and charities
(Local) government, charities and other public sector organisations increasingly interact with the public online, and an understanding of the nature of that interaction and the ways in which it may be analysed is potentially of enormous interest to policy-makers, e.g. in assessing levels of political engagement over time. Relevant case studies will be disseminated widely.

4. General public
Much of the data contained in the UK web archive is generated by members of the public, and this project will establish a research framework which will both safeguard their interests and help them to understand the uses to which their data may be put. A series of short videos will be produced, presenting the project's objectives and findings in an accessible way, one of which will discuss the ethics of big data research. The videos will be published on the project website and via YouTube. The project findings will also be disseminated using the well-established social media presences of the Institute of Historical Research, the BL and the Oxford Internet Institute.

5. Journalists and other mainstream media-based researchers
There is a substantial media interest in web-based and social media research, and this project will provide journalists with a greater understanding of the data itself and the context within which that research is conducted. This will in turn increase public understanding of both web archives and big data.