Residents ask follow-up ordinance questions in one long municipal code session. Which optimization keeps those turns fast?

Residents who ask follow-up ordinance questions in one long municipal code session create multi-turn context: prior clauses, section numbers, and clarifications must remain available without recomputing full attention from scratch each turn. Enabling key-value caching reuses prior-context keys and values so follow-ups stay fast while preserving conversational continuity—the standard inference optimization for civic chat over long code sessions. That reuse is why successive “what about the next subsection?” prompts do not re-pay the entire prompt tax. Recomputing full attention from scratch on every follow-up with no cache reuse correctly maintains context mathematically but wastes GPU time and inflates resident wait times during peak 311-like ordinance traffic. Flushing model weights from GPU memory between each resident question destroys residency benefits and forces cold-load latency unrelated to attention economics. Truncating the entire session history after every single token is generated annihilates the dialogue state the resident relies on and produces incoherent code advice. On the exam, KV-cache is the multi-turn latency answer for long civic sessions; distractors recompute everything, flush weights, or truncate history token-by-token. Caveat: unbounded cache growth on hour-long sessions can exhaust VRAM, so cities often pair KV-cache with windowing, summarization, or eviction policies rather than infinite retention. Operational check: on a scripted multi-turn ordinance dialogue, compare first-token and follow-up latency with KV-cache on versus off, confirm answer continuity across turns, and set a max-cache / window policy for the municipal code assistant.