2020 · 17 citations · 15 references
ComputingComputer SystemsEnvironmental SciencesBig Data AnalyticsEnvironmental MonitoringInternet Of ThingsData IntegrationData PipelinesDistributed Data ProcessingReal-time AnalyticsSensor NetworksStream ProcessingData Stream ProcessingDistributed Stream ProcessingEnvironmental IndicatorsEvent Stream ProcessingHeterogeneous Distributed SystemsPublish-subscribe SystemsReal-time ProcessingStreaming Data Processing
Distributed networks and real-time systems are becoming the most important components for the new computer age – the Internet of Things (IoT), with huge data streams generated from sensors and data sets generated from existing legacy systems. The data generated offers the ability to measure, infer and understand environmental indicators, from delicate ecologies and natural resources to urban environments. This can be achieved through the analysis of the heterogeneous data sources (structured and unstructured). In this paper, we propose a distributed framework – Event STream Processing Engine for Environmental Monitoring Domain (ESTemd) for the application of stream processing on heterogeneous environmental data. Our work in this area demonstrates the useful role big data techniques can play in an environmental decision support system, early warning and forecasting systems. The proposed framework addresses the challenges of data heterogeneity from heterogeneous systems and offers real-time processing of huge environmental datasets through a publish/subscribe method via a unified data pipeline with the application of Apache Kafka for real-time analytics.
15
Sanjay Ghemawat · Communications of the ACM · 2008 · 18.5K citations · Full text
Buğra Gedik, Henrique Andrade, Kun‐Lung Wu et al. · 2008 · 422 citations
Cayuga: A General Purpose Event Monitoring System.
Alan Demers, Johannes Gehrke, Biswanath Panda et al. · 2007 · 323 citations