Bigtable: a distributed storage system for structured data

TLDR

Bigtable is a distributed storage system for structured data that scales to petabytes across thousands of commodity servers and is used by many Google projects with diverse data sizes and latency requirements. The paper aims to describe Bigtable’s simple data model that gives clients dynamic control over data layout and format. It also details the design and implementation of Bigtable. Bigtable has successfully provided a flexible, high‑performance solution for all of these Google products despite the varied demands.

Abstract

Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.