Bigtable

TLDR

Bigtable is a distributed storage system designed to scale to petabytes across thousands of commodity servers, and it supports diverse Google applications such as web indexing, Google Earth, and Google Finance, each imposing distinct size and latency demands. The article aims to explain Bigtable’s simple data model that grants clients dynamic control over data layout and format. It details the design and implementation of Bigtable, outlining how the system achieves flexible, high-performance storage for varied workloads. Bigtable successfully delivers a flexible, high-performance solution that meets the diverse demands of all these Google products.

Abstract

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this article, we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.