Multi-site cooperative data stream analysis

Abstract

System S is a large-scale distributed streaming data analysis environment. Ultimately, we envision that there will be multiple sites running the System S software, each with their own administration and goals. However, cooperation between these sites can frequently be of mutual benefit. We are designing the framework to support numerous sites that can work both independently and in cooperative fashions, with a variety of interaction models such as peer-to-peer or federated. Depending on the degree of site autonomy and the relationships between any given pair of sites, the sites would be able to share data, perform processing on behalf of one another, or even take over tasks of a failed site. Interoperability is complicated by the degree of site autonomy as well as differences in execution environments and security policies. This paper surveys System S, describes its models for site interoperability, and discusses particular design issues such as site failover and heterogeneity.