Pergamum: replacing tape with energy efficient, reliable, disk-based archival storage

TLDR

Digital archival storage demands reliable, low‑power, cost‑effective solutions, yet tape systems lack random access and disk systems are energy‑intensive, leaving a gap for an efficient archival technology. Pergamum aims to provide a distributed, disk‑based archival system that is both reliable and energy‑efficient. Pergamum uses NVRAM at each node for signatures and metadata, enabling deferred writes and off‑disk verification, and employs intra‑ and inter‑disk redundancy with hash‑tree algebraic signatures and staggered rebuilds to reduce peak energy. Our evaluation shows Pergamum matches startup and ongoing costs of other archival technologies while delivering high reliability and adequate performance.

Abstract

As the world moves to digital storage for archival purposes, there is an increasing demand for reliable, low-power, cost-effective, easy-to-maintain storage that can still provide adequate performance for information retrieval and auditing purposes. Unfortunately, no current archival system adequately fulfills all of these requirements. Tape-based archival systems suffer from poor random access performance, which prevents the use of inter-media redundancy techniques and auditing, and requires the preservation of legacy hardware. Many disk-based systems are ill-suited for long-term storage because their high energy demands and management requirements make them cost-ineffective for archival purposes. Our solution, Pergamum, is a distributed network of intelligent, disk-based, storage appliances that stores data reliably and energy-efficiently. While existing MAID systems keep disks idle to save energy, Pergamum adds NVRAM at each node to store data signatures, metadata, and other small items, allowing deferred writes, metadata requests and inter-disk data verification to be performed while the disk is powered off. Pergamum uses both intra-disk and inter-disk redundancy to guard against data loss, relying on hash tree-like structures of algebraic signatures to efficiently verify the correctness of stored data. If failures occur, Pergamum uses staggered rebuild to reduce peak energy usage while rebuilding large redundancy stripes. We show that our approach is comparable in both startup and ongoing costs to other archival technologies and provides very high reliability. An evaluation of our implementation of Pergamum shows that it provides adequate performance.