RAnDMS (Real time Analysis of Digital Media Streams)

RAnDMS will study, implement and evaluate Real-time Data and Visual Analytic techniques to enable intelligence agencies, the MoD, the police and emergency responders to monitor and make sense of local, regional and global events using web-scale data from social and traditional media streams. The intelligence gathering task will be defined as identifying, correlating, integrating and presenting data and information, in order to understand situations as they arise. Current technology does not provide efficient and effective solutions, as it mainly focuses on detecting trends in the use of keywords and tags. While this is able to spot overall patterns in the data, it just enables the retrieval of relevant documents, without any correlation and integration of the contained information. Moreover, information concerning local situations and events, which may only be discussed within a handful of documents, is ignored.
Within RAnDMS data analytics will focus on enabling the capture of information from media streams; illuminating situations at all levels, from global to local. This information will support decision making for the intelligence community, which is expected to increase their ability to monitor events and situations relevant to homeland security and to peace-keeping efforts. The scientific challenge is that data and information in these streams are: (i) high in volume, and constantly increasing, (ii) often duplicated, incomplete, imprecise and incorrect; (iii) written in informal style (i.e. short, unedited and conversational); and (iv) generally concerning the short-term zeitgeist. These characteristics make analysis very hard, especially when considering that major requirements of the intelligence community are that (i) documents must be processed in real-time and (ii) the relevant information may be in the long-tail of the distribution, i.e. it may be mentioned very infrequently.
We will provide highly efficient and effective technologies able to associate each document with its context. A documents context is provided by four dimensions: (who) the author of the document, (when) the time it was sent, (where) the location referred to in the document and (what) other documents with similar content. This information is either provided by the media stream or extracted from the document's content using efficient statistical text-mining techniques. By interpreting documents in terms of these four dimensions we enable: (i) the detection of events, i.e. documents and their content (what) are clustered around a time and place; (ii) the profiling of authors from the content (what and where) of the documents they have created; and (iii) determine information that is missing or ambiguous in document, using information present in the documents within their context.
Visual analytics will facilitate the exploration of the information by providing multiple views; enabling focused investigation and trend visualisations across the four dimensions. We will devise methods to (i) suggest the right level of detail (granularity) for the user focus in rapidly changing environments; (ii) alert users to any significant development outside of their current viewpoint; and (iii) enable users to understand how the current state of affairs came into being by browsing along the all information along the time dimension. Methods will en able to see through the irrelevant banter (noise) that often surround events in social media and go directly to the relevant information that can be hidden in the long tail of the distribution.
RAnDMS will be tested on the task of supporting intelligence operators during relevant events happening during 2012/13. We will publish the research and its findings in international journal and conferences. Subject to MoD agreement, we will also create public research resources by generating one publicly available task (inclusive of corpora, resources, etc.) to enable comparison of research results by other researchers.

The project will have an impact on industry, the government and society.

The main target of dissemination and exploitation will be companies working in the fields of intelligence/anti-terrorism, such as Ultra Electronics (one of the major providers of the UK government) both in the UK and abroad. These companies tend to have a traditional focus and are not currently working on solutions involving the use of the social Web to help make sense of events. This is despite the large interest created at an international level by events such as the Arab Spring and Hurricane Katrina and the large interest in the intelligence and defence sector, as demonstrated by some recent NATO reports, especially of US origin.

We will build on technology and know-how developed as part of the WeKnowIt and TRIDS projects, moving towards a level of complexity that currently is not possible to reach with that technology, as some fundamental research problems must be addressed. We will develop the science and transfer some intermediate results on top of the baseline technology. Moreover, we will create demonstrators of capabilities that are made possible by the research but that will need more investments (e.g. via TSB or MoD funded projects) to create actual applications.

It is expected that the technologies will be industrialised and applied in subsequent projects through collaboration with companies. Exemplars of datasets will be made available for benchmarking and evolving technology in this field and for future experiments. We have a long and successful experience in collaborating with industry and we expect this to continue with this project.

As for the government, the science and technology developed in the project will provide a feasibility study in the use of social Web monitoring for intelligence. The technology developed in the project will enable a deep insight into crises and emergencies nationally and abroad. Recent events, both local (e.g. the Aug 2011 UK riots), but especially abroad (e.g. during the Arab Spring) have shown that Western governments struggled to make sense of the events as they were missing direct information from the ground. The technology proposed in this project will enable harvesting of information directly from the ground, via the monitoring of social media and news feeds. We plan to test the developed technology on corpora collected during emergencies and crises which develop during 2012. This will show the way to how future emergencies and crises could be managed, hence creating an interest in the government on the exploitation of the results. DSTL will be key to this plan: We plan to spend several weeks at DSTL to derive requirements, to co-create the science and the technology and finally to test the results in real world scenarios with real users. The time frame for exploitation of results will be 1-5 years after the project end, when products based on the developed technology will be available.

Finally citizens and society will benefit from the results of the project in an indirect way through improved security nationally and internationally. More directly, the ability to gain intelligence through the use and analysis of social media may be reused by the ordinary citizen in, for example, education and to encourage civic engagement.