DISSP: Dependable Internet-Scale Stream Processing

Real-time stream data has begun to play an increasingly important role on the Internet. One of the causes for this is the proliferation of geographically-distributed stream data sources such as sensor networks, scientific instruments, pervasive computing environments and web feeds connected to the Internet. Potentially millions of users world-wide want to take advantage of the availability of this data. Therefore they require a convenient way to process real-time stream data at a global scale through applications that perform Internet-scale stream processing (ISSP). Analogous to how search engines make static web data useful to users, an ISSP system reliably collects, filters and processes stream data from potentially thousands of data sources on behalf of many users.The research focus of this proposal is to address a major challenge facing all ISSP applications: achieving robustness in the presence of failures. Failures are a fact of life in such systems because of their scale, heterogeneity and reliance on best-effort Internet infrastructure. This conflicts with the requirements of many users that demand a dependable ISSP (DISSP) service --- a service that should continue functioning even during Internet path outages, processing host failures and resource shortages. For example, consider a scientific study that detects and analyses transient events in the sky by correlating high-bandwidth, real-time image streams from dozens of radio telescopes world-wide. After the failure of the network connection to one of the telescopes, a scientist would expect the system to continue operating, perhaps with reduced resolution. Similarly, the failure of a host that processes the sky images should not cause a service interruption, although it may lead to a decrease in detection confidence of anomalies. Unfortunately, mechanisms for building DISSP systems are lacking. Conventional techniques for reliable data processing cannot be applied to a DISSP system because of its requirements of global scalability, of short recovery times in a real-time setting and of resource efficiency due to a shared infrastructure.This proposal addresses these challenges so that ISSP systems needed for interconnecting tomorrow's pervasive sensor systems and global scientific experiments can become a reality. We intend to develop new techniques for building dependable ISSP systems, taking their unique features in terms of scale, failure model and data quality into account. In particular, we will devise approaches that gracefully degrade result quality in response to resource shortage after failure. While result quality is reduced, the system will provide constant feedback to users on the achieved level of service. Feedback will be expressed in a domain-specific way, e.g., by notifying a scientific user about the reduction in detection confidence of events of interest. This feedback will also drive an adaptive fault-tolerance mechanism, allowing the DISSP system to strategise about resource allocation in order to minimise the reduction in service quality of a maximum number of users.