Manipulation and extraction of knowledge from large datasets with desktop tools optimised for the analysis of 'Big Data'

We live in a 'data tsunami' and there is often too much data to handle without specialist skills and this is especially true for analysts (e.g. volunteers or small organisations) with limited resources. Large systems and/or communities of interest have the potential to generate huge amounts of data that can be kept almost indefinitely in low cost digital archives. There are also activities converting historical data from many different domains into searchable digital formats, sometimes by automated processes, and other times by volunteer effort. For example, citizen science data assembled from diverse records that go back decades is used to analyse and understand biodiversity; this data is assembled and analysed almost entirely by volunteer enthusiasts. The ability to effectively analyse Big Data with tools designed for desktop computers (or equivalent) has not kept pace with the ability to generate it. Familiar techniques, such as spreadsheets, do not scale effectively as the size of the dataset increases beyond that which can be checked by hand. Furthermore, programming languages for analysing large datasets with limited resources rely on the technical expertise of the programmer to write efficient code that will execute quickly. Currently there is no effective method to encapsulate that expertise into a template that can be reused on related sets of data. The reliability of such datasets may be compromised by inconsistent data cleansing techniques, which all rely (in any case) on knowing an expected form and range of the data; this is not a valid assumption for some datasets.
This work will develop a novel conceptual framework using task orientated templates, for the analysis of Big Data that effectively separates: data curation, analysis, and reporting of large datasets, while creating, a reproducible analysis. Outputs should include secondary data, with metadata, that captures all the transformations that have been applied, to provide an auditable connection to the source data. The objective is to support the reuse of secondary data with confidence in downstream analysis. Using task orientated templates should allow a more structured approach to data analysis, and facilitate reuse of data through the use of verifiable digital signatures. The template-based approach is intended to reduce the programming skills required for the analysis large data for a wide range of commercial, academic and social applications on desktop computers (or equivalent).
Fundamentally, this research will challenge the sequential nature of data handling (load, transform, analyse, output), to reduce the risk of irreversible load/transform/analysis- induced error propagation to enable users with limited IT resources and moderate algorithmic and programming skills to reliably extract new knowledge from large and complex datasets.
The relevant EPSRC sub theme is Data Information and knowledge