eScholarship (California Digital Library) · 2016 · 61 citations · 11 references
Open access
ComputingComputer ScienceComputer SystemsDatabasesInternet Of ThingsStorage SystemsCluster Storage SystemsData Storage SystemsDatabase ManagementQuery ProcessingStream ProcessingTelemetry SystemsAbstractionCompression RatioCopy-on-writeCopy-on-write StorageData-intensive ComputingTelemetry Data CompressionTelemetry StorageThroughput
The increase in high-precision, high-sample-rate telemetry timeseries poses a problem for existing timeseries databases which can neither cope with the throughput demands of these streams nor provide the necessary primitives for effective analysis of them. We present a novel abstraction for telemetry timeseries data and a data structure for providing this abstraction: a timepartitioning version-annotated copy-on-write tree. An implementation in Go is shown to outperform existing solutions, demonstrating a throughput of 53 million inserted values per second and 119 million queried values per second on a four-node cluster. The system achieves a 2.9x compression ratio and satisfies statistical queries spanning a year of data in under 200ms, as demonstrated on a year-long production deployment storing 2.1 trillion data points. The principles and design of this database are generally applicable to a large variety of timeseries types and represent a significant advance in the development of technology for the Internet of Things.
11
Ceph: a scalable, high-performance distributed file system
Sage A. Weil, Scott Brandt, Ethan L. Miller et al. · eScholarship (California Digital Library) · 2006 · 1.4K citations · Full text
Matt Welsh, David Culler · 2001 · 847 citations
Tuomas Pelkonen, Scott Franklin, Justin Teller et al. · Proceedings of the VLDB Endowment · 2015 · 283 citations
Michael Stonebraker, Ariel Weisberg · 2013 · 216 citations