Proceedings of the ACM on Management of Data · 2023 · 18 citations · 24 references
ComputingComputer ScienceComputer SystemsBig Data AnalyticsData ManagementDatabasesCaching StrategiesDistributed Data ProcessingLatency OptimizationQuery OptimizationQuery ProcessingContainer Resource AllocationElastic Resource AllocationEtl ProcessingHierarchical CachingMachine Learning WorkloadsQuery EvaluationQuery Latency OptimizationQuery Performance Optimization
Presto is an open-source distributed SQL query engine that supports analytics workloads involving multiple exabyte-scale data sources. Presto is used for low-latency interactive use cases as well as long-running ETL jobs at Meta. It was originally launched at Meta in 2013 and donated to the Linux Foundation in 2019. Over the last ten years, upholding query latency and scalability with the hyper growth of data volume at Meta as well as new SQL analytics requirements have raised impressive challenges for Presto. A top priority has been ensuring query reliability does not regress with the shift towards smaller, more elastic container allocation, which requires queries to run with substantially smaller memory headroom and can be preempted at any time. Additionally, new demands from machine learning, privacy, and graph analytics have driven Presto maintainers to think beyond traditional data analytics. In this paper, we discuss several successful evolutions in recent years that have improved Presto latency as well as scalability by several orders of magnitude in production at Meta. Some of the notable ones are hierarchical caching, native vectorized execution engines, materialized views, and Presto on Spark. With these new capabilities, we have deprecated or are in the process of deprecating various legacy query engines so that Presto becomes the single piece to serve interactive, ad-hoc, ETL, and graph processing workloads for the entire data warehouse.
24
2017 USENIX Annual Technical Conference (USENIX ATC'17)
Do Le Quoc, Martin S. Beck, Pramod Bhatotia et al. · 2017 · 3.1K citations
Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas et al. · 2013 · 1.8K citations
Michael Armbrust, Reynold Xin, Cheng Lian et al. · 2015 · 1.2K citations
Hive - a petabyte scale data warehouse using Hadoop
Ashish Thusoo, Joydeep Sen Sarma, Namit Jain et al. · 2010 · 925 citations
Sergey Melnik, Andrey Gubarev, Jing Jing Long et al. · Proceedings of the VLDB Endowment · 2010 · 709 citations