Understanding the annotation process: annotation for Big data

Data is being collected and created at the fastest rate in human history; by the far the vast majority of this is in digital format. Allied with this, what was previously "offline" information can now be digitised quickly and cheaply e.g. old manuscripts, maps etc. This vast collection of existing and new information creates new opportunities and also difficulties. For a lot of this information to be useful it must be categorised and annotated in some way, so that sense can be made of the data and also so that the correct data can be accessed more easily. It is possible to complete this categorisation by hand with human annotators, but this effort can be expensive in terms of time, money and resources. This is especially true for large data sets or for data sets that require niche expertise to annotate. With this expense in mind, many have turned to machine learning to annotate data; however machine learning approaches still require human intervention to both create training sets for algorithms and judge the output of algorithms. Thus it is inevitable that human intervention is involved at some stage of the categorisation and annotation process. In this project we aim to gain a better understanding of this annotation process so that we can provide guidelines, approaches and processes for providing the most cost effective and accurate annotations for data sets.

We propose to work with the three main types of unstructured data faced in big data: text, image, and video. The first challenge is to better understand the process assessors go through when annotating and judging different types of material. This will be carried out using a mixture of qualitative and quantitative techniques, using smaller scale lab-based studies. By better understanding the process by which individuals annotate and classify material, we hope to provide insights which can be used to make the annotation process more efficient, and identify a set of initial factors which affect annotation performance, such as degree of domain expertise and time. Based on this initial work, the aim is to then investigate which of these factors most affect assessment, using large scale crowdsourcing style methods. The final challenge is related to the classification task: how should annotation be approached, to give the best results when used in machine learning? Based on this, the project aims to create a set of guidelines for the creation of annotation and relevance sets.

We expect the project to have impact both within the research community, and outside. We have identified the following groups which will be the primary beneficiaries for our work:
- The research community - the project will generate a annotation sets which can be used in a range of future research, including the evaluation of novel machine learning systems (e.g. for classification) and information retrieval systems. Outside these technical areas, there is also potential impact in the general social science and information science area, by providing guidelines which can be used for the creation of annotation sets, uniquely tailored to the new automatic learning systems designed for big data.
- Digital librarians and archivists- the availability of best practises for annotating digital archives will make it easier for librarians and archivists to prepare for the automatic "big data" technologies. Automatic annotation is still in it's infancy, rarely applied in end-user focused repositories, either in industry or in academia. Guidelines which both maximise the automatic techniques, and demonstrate what the techniques are capable of, and perhaps even more importantly, what automatic techniques are not capable of, is expected to be valuable for information professionals. Ultimately, automated systems will aid individuals to maintain and share digital archives, which will be especially pertinent for specialised archives where there is a lack of resources for manual classification e.g. niche historical archives.
- Industry information professionals - similar to the situation above for digital librarians and archivists, many information professionals are currently under great cost and time pressures, e.g. health information librarians, or legal research. In such very specialised domains, access to experts may be very expensive e.g. in the legal domain, paying experienced layers to judge the relevance of material to cases can be very costly. Ultimately the annotation design guidelines which would be created in this project would allow organisations to focus sparse resources in particular areas, in order to maximise the performance of automated classification techniques, and minimise the use of experts (e.g. using experts only for marginally relevant or different to categorise material).
- Educators - similar to the case of research community, the project will generate data sets which can also be used in education. We would also hope that such open data sets would encourage the use of automatic techniques outside the research community, since the resources will be more easily available to students and teachers alike.
- General public -It is anticipated that through this work that digital archives can be annotated more quickly and efficiently. In addition it is hoped that physical resources and archives can be digitised and annotated more quickly and efficiently. Thus, ultimately, more of these resources will become available to the UK population. Access to these resources will be beneficial in terms of learning and awareness, as well as providing access to information that otherwise would not be available to the public or indeed may even have been lost.