2025 · 23 citations · 13 references
ComputingComputer EngineeringComputer ScienceMemoryParallel ComputingVirtualizationCachingMemory HierarchyMemory Hierarchy OptimizationMemory ManagementMemory Subsystem DesignVirtual MemoryCache DesignDynamic Memory AllocationDynamic Memory ManagementFragmentationMemory AllocationMemory PlacementVirtual Memory ManagementVirtual Memory Systems
PagedAttention is a popular approach for dynamic memory allocation in LLM serving systems. It enables on-demand allocation of GPU memory to mitigate KV cache fragmentation - a phenomenon that crippled the batch size (and consequently throughput) in prior systems. However, in trying to allocate physical memory at runtime, PagedAttention ends up changing the virtual memory layout of the KV cache from contiguous to non-contiguous. Such a design leads to non-trivial programming and performance overheads.
13
Advances in Neural Information Processing Systems 29
Onur Teymur, Kostas Zygalakis, Ben Calderhead · 2016 · 13.5K citations
Proceedings of the 24th international conference on Machine learning
2007 · 11.7K citations
PaLM: Scaling Language Modeling with Pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin et al. · arXiv (Cornell University) · 2022 · 2.1K citations · Full text
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Z. Li, Siyuan Zhuang et al. · 2023 · 1.1K citations · Full text