BetterCrowd: Human Computation for Big Data

In the last few years we have seen a rapid increase of available data. Digitization has become endemic. This has lead to a data deluge that left many unable to cope with such large amounts of messy data. Also because of the large number of content producers and different formats, data is not always easy to process by machines due to its its diverse quality and the presence of bias. Thus, in the current data-driven economy, if organizations can effectively analyze data at scale and use it as decision-support infrastructure at the executive level, data will lead to a key competitive advantage. To deal with the current data deluge, in the BetterCrowd project I will define and evaluate Human Computation methods to improve both the effectiveness and efficiency of currently available hybrid Human-Machine systems.

Human Computation (HC) is a game-changing paradigm that systematically exploits human intelligence at scale to improve purely machine-based data management systems (see, for example, CrowdDB [13]). This is often obtained by means of Crowdsourcing, that is, outsourcing certain tasks from the machine to a crowd of human individuals who perform short tasks (also known as Human Intelligence Tasks or HITs) that are simple for humans but still difficult for machines (e.g., understanding the content of a picture or sarcasm in text). Involving humans in the computation process is a fundamental scientific challenge that requires obtaining the best from human abilities and effectively embedding them into traditional computational systems. The challenges involved with the use of HC are both its efficiency (i.e., humans are naturally slower than machines in terms of information processing) and effectiveness (i.e., while machines deterministically compute, humans behavior may be unpredictable and possibly malicious).

The project is composed of two main parts. We will first look at how to improve crowdsourcing effectiveness by proposing novel techniques to detect malicious workers in crowdsourcing platforms. In the second part, we will make HC techniques scale so that they can be applied to larger volume of data focusing on scheduling tasks to the crowd (WP2).

Other than academic beneficiaries, this research will impact commercial organizations and people involved in crowdsourcing activities.

Big Data Analytics Market.
The scale of data currently being produced by large organizations requires novel ways of managing and, most importantly, analyzing such enormous amounts of data in order to produce value for data consumers (e.g., company customers, employees, or governmental organization clients). In this context, high data quality is critical. The techniques developed within the scope of this project for efficient and effective human computation can be used to create better Big Data solutions and products in coordination with analytics platforms used within large-scale organizations. An example of industry use of hybrid human-machine techniques by means of crowdsourcing is already in place at Twitter where new trending topics and acronyms are detected in real-time by a mix of machine-based stream processing approaches and crowdsourcing on Amazon Mechanical Turk. The outcome of the BetterCrowd project has potential for impact on Big Data crowd-based solutions by optimizing requests to the crowd and the overall output quality produced.

Enterprise crowdsourcing.
In the enterprise domain, large companies (e.g., IBM, Microsoft, VeriSign) have already started to run in-house crowdsourcing: Employees of the company are the crowd to which data quality HITs are sent. Crowdsourcing in this context is different in aspects such as worker reputation and incentives. However, research findings on scalability aspects will be directly applicable to this way of leveraging knowledge workers within companies. Scheduling HITs to a crowd for in-house crowdsourcing is extremely important as this can be considered as a single multi-tenant system with jobs run with different priorities.

Crowdsourcing as career path.
The crowdsourcing market has seen an exponential growth over the last few years with a doubling of the market over the last two years. That said, crowdsourcing is still a highly unregulated market. In the longer term, the more efficient and effective use of crowdsourcing that will result from this project will support the creation of better conditions for the work of the crowd not excluding the definition of crowd work as a profession which is already a reality in developing countries such as India.