2020 · 12 citations · 17 references
ComputingComputer EngineeringComputer ScienceFault ToleranceFault-tolerant ComputingFault-tolerant SystemsFault Tolerance DesignFault Tolerance In Parallel SystemsFault Tolerance OptimizationFault-tolerant SchedulingLatency OptimizationStream ProcessingCheckpoint Interval OptimizationCheckpointingCheckpointing OptimizationCheckpointsFault Tolerance MechanismsParallel ProfilingPerformance ObjectivesProcessing Latency
Fault tolerance is a property which needs deeper consideration when dealing with streaming jobs requiring high levels of availability and low-latency processing even in case of failures where Quality-of-Service constraints must be adhered to. Typically, systems achieve fault tolerance and the ability to recover automatically from partial failures by implementing Checkpoint and Rollback Recovery. However, this is an expensive operation which impacts negatively on the overall performance of the system and manually optimizing fault tolerance for specific jobs is a difficult and time consuming task.In this paper we introduce Chiron, an approach for automatically optimizing the frequency with which checkpoints are performed in streaming jobs. For any chosen job, parallel profiling runs are performed, each containing a variant of the configurations, with the resulting metrics used to model the impact of checkpoint-based fault tolerance on performance and availability. Understanding these relationships is key to minimizing performance objectives and meeting strict Quality-of-Service constraints. We implemented Chiron prototypically together with Apache Flink and demonstrate its usefulness experimentally.
17
Spark: cluster computing with working sets
Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin et al. · 2010 · 4.2K citations
Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas et al. · 2013 · 1.8K citations
Large-scale cluster management at Google with Borg
Abhishek Verma, Luis Pedrosa, Madhukar Korupolu et al. · 2015 · 1.3K citations · Full text
Ankit Toshniwal, Siddarth Taneja, Amit K. Shukla et al. · 2014 · 858 citations