Towards scalable real-time analytics

TLDR

We present an overview of the SAP HANA Scale‑out Extension, a novel distributed database architecture designed to support large‑scale analytics over real‑time data and to meet modern enterprise needs. The architecture decouples core database components—query processing, concurrency control, and persistence—leveraging high‑throughput, low‑latency networks and storage, provides full ACID guarantees with MVCC‑based snapshot isolation via logical timestamps, and achieves consistency through asynchronous update propagation. The platform delivers high‑performance OLAP at massive scale‑out while supporting concurrent OLTP workloads, enabling analytics on continuously changing data with fine‑grained service‑level agreements on data freshness.

Abstract

We present an overview of our work on the SAP HANA Scale-out Extension, a novel distributed database architecture designed to support large scale analytics over real-time data. This platform permits high performance OLAP with massive scale-out capabilities, while concurrently allowing OLTP workloads. This dual capability enables analytics over real-time changing data and allows fine grained user-specified service level agreements (SLAs) on data freshness. We advocate the decoupling of core database components such as query processing, concurrency control, and persistence, a design choice made possible by advances in high-throughput low-latency networks and storage devices. We provide full ACID guarantees and build on a logical timestamp mechanism to provide MVCC-based snapshot isolation, while not requiring synchronous updates of replicas. Instead, we use asynchronous update propagation guaranteeing consistency with timestamp validation. We provide a view into the design and development of a large scale data management platform for real-time analytics, driven by the needs of modern enterprise customers.