2024 · 33 citations · 6 references
ComputingComputer EngineeringComputer ScienceHeterogeneous ComputingParallel AlgorithmsParallel ComputingCompute Resource ManagementData ParallelismMemory Hierarchy OptimizationParallel ArchitecturesParallel Computing SystemsParallel Performance AnalysisCommunication EfficiencyEfficiencyHigh-performance Computing SystemsKey-value Cache ManagementLarge Language ModelsLong-context Language ModelingLong-context ModelingParallel Performance Optimization
The context window of large language models (LLMs) is rapidly increasing, leading to a huge variance in resource usage between different requests as well as between different phases of the same request. Restricted by static parallelism strategies, existing LLM serving systems cannot efficiently utilize the underlying resources to serve variable-length requests in different phases. To address this problem, we propose a new parallelism paradigm, elastic sequence parallelism (ESP), to elastically adapt to the variance across different requests and phases. Based on ESP, we design and build LoongServe, an LLM serving system that (1) improves computation efficiency by elastically adjusting the degree of parallelism in real-time, (2) improves communication efficiency by reducing key-value cache migration overhead and overlapping partial decoding communication with computation, and (3) improves GPU memory efficiency by reducing key-value cache fragmentation across instances. Our evaluation under diverse real-world datasets shows that LoongServe improves the throughput by up to 3.85× compared to chunked prefill and 5.81× compared to prefill-decoding disaggregation.
6
Yanghua Peng, Yixin Bao, Yangrui Chen et al. · 2018 · 399 citations
Efficient dynamic programming using quadrangle inequalities
F. Frances Yao · 1980 · 136 citations · Full text
Sequence Parallelism: Long Sequence Training from System Perspective
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal et al. · 2023 · 29 citations · Full text