During the inference phase of a large language model (LLM), the Key-Value (KV) cache grows dynamically and can quickly deplete GPU VRAM. To resolve this, which memory management technique allocates KV cache fragments non-contiguously in memory using virtual-to-physical mapping, drastically minimizing fragmentation and wasting zero memory?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Now, pay close attention here, because this memory bottleneck is a huge deal for LLMs. When you generate text, you save old keys and values in memory so you don't have to recalculate them. But if you try to allocate a huge, continuous block of GPU memory for every request, you end up with massive fragmentation—what we call memory waste. The cool solution here is PagedAttention (popularized by vLLM). Think of it just like the virtual memory paging in your computer's OS. It breaks the KV cache into small blocks and maps them dynamically. No need for continuous memory, and zero fragmentation! FlashAttention is awesome for speed because it optimizes how data moves between GPU caches, but PagedAttention is the champion for managing dynamic memory allocation.
Full explanation below image
Full Explanation
In Large Language Model (LLM) serving, the Key-Value (KV) cache stores past keys and values during autoregressive decoding to avoid redundant computations. However, because sequence lengths vary and grow dynamically, allocating static, contiguous memory blocks for the KV cache leads to significant GPU memory inefficiencies. Up to 60-80% of memory can be wasted due to internal fragmentation (pre-allocating space for the maximum possible sequence length) and external fragmentation.
PagedAttention solves this issue by borrowing the concept of virtual memory and paging from operating systems. Instead of requiring contiguous memory allocation for the KV cache of a sequence, PagedAttention partitions the KV cache into fixed-size physical blocks. The system maintains a lookup table that maps virtual blocks (representing sequence tokens) to non-contiguous physical blocks in the GPU's High Bandwidth Memory (HBM). This virtual-to-physical block mapping allows physical memory to be allocated on-demand, reducing memory waste to near zero and enabling higher batch sizes and throughput.
Let's review the other options: - FlashAttention (Option A) is a hardware-aware attention algorithm that optimizes training and inference speed by using tiling to minimize memory reads/writes between GPU High Bandwidth Memory (HBM) and SRAM. It focuses on computational speed and memory IO bandwidth, not dynamic KV cache memory allocation. - Tensor Parallelism (Option C) splits individual tensor operations (like weight matrices) across multiple GPUs, which helps fit large models into memory but does not solve the dynamic KV cache fragmentation problem within a GPU's memory space. - Standard KV Caching with linear projection (Option D) is a general design for caching but does not address the physical memory layout fragmentation issues solved by paging.