# How LLM Inference Actually Runs: Tokens, KV Cache, and Why It's Memory-Bound > A language model generates one token at a time, and each token re-reads gigabytes of weights and a growing cache. Understanding the KV cache and the memory-bandwidth wall explains latency, context limits, and every serving optimization from the last two years. URL: https://harshith.in/blog/how-llm-inference-works/ Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/ Tags: ai, performance, systems I came to LLMs from the systems side, and the framing that finally made them legible wasn't the math of attention — it was treating inference as what it is: a memory-bound serving workload with an unusual cache. Once you see that a model generating text is *bandwidth-limited, not compute-limited*, the whole landscape of confusing behavior (why the first token is slow but the rest are steady, why context length costs so much memory, why batching is everything) falls into place. This is that systems-eye view. ## Generation Is a Loop, and Each Step Re-Reads the Model An autoregressive LLM produces one **token** at a time (tokens are sub-word chunks — a tokenizer splits text into ~4-characters-ish pieces from a fixed vocabulary). Each step: feed the whole sequence so far through the network, get a probability distribution over the vocabulary, sample one token, append it, repeat. ```text "The capital of France is" → [forward pass] → "Paris" (prob 0.8...) "The capital of France is Paris" → [forward pass] → "." → ... ``` The systems-critical fact: **every single token requires reading all of the model's weights.** A 70-billion-parameter model at 16-bit is 140 GB; at 4-bit quantization ~35 GB. To generate one token, the hardware streams those tens of gigabytes from GPU memory through the compute units. To generate 500 tokens, it does that 500 times. The arithmetic per token is modest; the *data movement* is enormous — which means inference speed is gated by **memory bandwidth**, not FLOPs. A GPU rated for petaflops sits mostly idle during single-stream decode, waiting on memory. This one fact is why an H100's headline compute number tells you almost nothing about its tokens-per-second. ## The KV Cache: The Optimization Everything Depends On Naively, re-processing the entire growing sequence each step is O(n²) — token 500 reprocesses 499 predecessors. The fix that makes inference tractable is the **KV cache**: attention computes a "key" and "value" vector per token, and *those don't change* once computed. So cache them; each new token computes its own K/V and attends over the cached ones, turning per-step work from "reprocess everything" to "process one new token against the cache." This splits inference into two phases with completely different performance characteristics — the distinction behind every latency number you'll see quoted: - **Prefill**: process the entire prompt at once, building the KV cache. Parallel across all prompt tokens, compute-heavy, fast per token. This is your **time-to-first-token**, and it scales with prompt length. - **Decode**: generate tokens one at a time, each reading the whole model + the whole KV cache. Sequential, memory-bound, one token per forward pass. This sets your **tokens-per-second**. "Long prompts are slow to start, then generate steadily" is exactly this: a big prefill, then bandwidth-limited decode. The KV cache also explains the **context-length memory wall**. The cache grows linearly with sequence length and is often the dominant memory consumer at long context — potentially many gigabytes for one long conversation, *per request*. This is why long context is expensive, why serving frameworks obsess over it, and why **PagedAttention** (the idea behind vLLM, ~2023) mattered so much: it manages the KV cache in fixed pages like an OS manages virtual memory, ending the fragmentation that used to waste most of it — a page table for attention, which is a deeply satisfying convergence for anyone who's read this blog's OS posts. ## Why Serving Looks the Way It Does Every production inference optimization is an answer to "we're memory-bandwidth-bound, so amortize the weight reads": - **Batching is the core lever.** Reading the weights to serve one request costs the same as reading them to serve fifty *concurrent* requests — the weights stream once, the matmul does more work per byte read. So throughput scales dramatically with batch size until compute or KV-cache memory becomes the limit. **Continuous batching** (swap finished sequences out and new ones in each step, rather than waiting for a whole batch to finish) is why modern servers hit high utilization. This is also the fundamental tension: latency (small batch, dedicated bandwidth) vs throughput/cost (big batch). - **Quantization** (16-bit → 8/4-bit weights) directly attacks the bottleneck: fewer bytes per weight = less to stream = faster decode and less memory, at some accuracy cost. It's a bandwidth optimization first, a memory one second. - **Speculative decoding** breaks the one-token-per-pass sequential floor: a small cheap model drafts several tokens, the big model *verifies* them in one parallel pass, accepting the run that matches. Multiple tokens per expensive forward pass when the draft is good. None of this requires understanding transformers deeply — it requires understanding that you're moving a lot of memory repeatedly, which is the most systems-engineering problem imaginable. Profiling LLM serving is bandwidth accounting, queueing, and cache management: familiar tools pointed at a new workload. ## Takeaways - LLMs decode one token at a time, re-reading all model weights per token — inference is memory-bandwidth-bound, so GPU FLOP ratings mislead. - The KV cache turns O(n²) decode into per-token work by caching unchanging key/value vectors; prefill (TTFT) and decode (tok/s) are separate performance regimes. - KV cache grows with context length and is the long-context memory wall; PagedAttention manages it like OS virtual memory. - Batching amortizes the fixed weight-read cost across requests — continuous batching is why servers stay utilized; latency vs throughput is the central trade. - Quantization cuts bytes-per-weight (bandwidth), speculative decoding cuts passes-per-token — both attack the real bottleneck.