2025 · 15 citations · 3 references
ComputingComputer ScienceHigh-performance ComputingMachine LearningNatural Language ProcessingParallel ComputingData ParallelismGraphics Processing Unit ComputingLanguage ModelingMemory Hierarchy OptimizationParallel ArchitecturesParallel Computing SystemsGpu ArchitectureKernel OptimizationLarge Language ModelsModel ParallelismParallel InferenceParallel Machine LearningQuantizationWorkload
As inference on Large Language Models (LLMs) emerges as an important workload in machine learning applications, model weight quantization has become a standard technique for efficient GPU deployment. Quantization not only reduces model size, but has also been shown to yield substantial speedups for single-user inference, due to reduced memory movement, with low accuracy impact. Yet, it remains a key open question whether speedups are achievable also in batched settings with multiple parallel clients, which are highly relevant for practical serving. It is unclear whether GPU kernels can be designed to remain practically memory-bound, while supporting the substantially increased compute requirements of batched workloads.
3
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Z. Li, Siyuan Zhuang et al. · 2023 · 1.1K citations · Full text
Muhammad Osama, Duane Merrill, Cris Cecka et al. · 2023 · 17 citations