Starfish: A Self-tuning System for Big Data Analytics.

TLDR

Timely and cost‑effective analytics over Big Data is essential for businesses, science, engineering, and government, yet most practitioners lack the expertise to tune Hadoop, whose out‑of‑the‑box performance is suboptimal, leading to wasted resources and time. We introduce Starfish, a self‑tuning system for big data analytics. Starfish builds on Hadoop, automatically adapting to user needs and workloads to deliver good performance without requiring users to manipulate tuning knobs, and its architecture, guided by self‑tuning database research, incorporates design choices that address the unique challenges of modern big‑data analysis practices.

Abstract

Timely and cost-effective analytics over “Big Data” is now a key ingredient for success in many businesses, scientific and engineering disciplines, and government endeavors. The Hadoop software stack—which consists of an extensible MapReduce execution engine, pluggable distributed storage engines, and a range of procedural to declarative interfaces—is a popular choice for big data analytics. Most practitioners of big data analytics—like computational scientists, systems researchers, and business analysts—lack the expertise to tune the system to get good performance. Unfortunately, Hadoop’s performance out of the box leaves much to be desired, leading to suboptimal use of resources, time, and money (in payas-you-go clouds). We introduce Starfish, a self-tuning system for big data analytics. Starfish builds on Hadoop while adapting to user needs and system workloads to provide good performance automatically, without any need for users to understand and manipulate the many tuning knobs in Hadoop. While Starfish’s system architecture is guided by work on self-tuning database systems, we discuss how new analysis practices over big data pose new challenges; leading us to different design choices in Starfish.