Monitoring and Debugging DryadLINQ Applications with Daphne

Abstract

Debugging and optimizing large-scale applications is still more art than engineering discipline. This document describes our experience in building a set of tools to help DryadLINQ application developers understand and debug their programs. The core infrastructure for our tools is a portable library which provides a DryadLINQ job object model (i.e., a local representation of the distributed state of an executed application). Layered on the job object model we have built a variety of interactive and batch tools for: performance data collection and analysis, distributed state visualization, failure diagnostics, debugging, and profiling.