A water utility chatbot streams tokens to residents and needs low first-token latency. Which combination helps?

A water utility chatbot that streams tokens to residents is judged first by time-to-first-token and then by steady decode speed for boil-water, outage, and billing follow-ups. Combining a TensorRT runtime with key-value caching addresses both: the engine accelerates the compiled graph while KV-cache reuses prior states so streaming follow-ons stay cheap—matching civic streaming UX goals. That pairing is a standard production recipe when GPUs and engine support are available. Serializing every resident reply to magnetic tape before the first token appears invents archival latency incompatible with live chat. Disabling caching and running only uncompiled eager Python for each streamed token forgoes both major latency levers. Buffering the full answer offline overnight and never streaming tokens live abandons the streaming product requirement entirely. On the exam, TensorRT plus KV-cache is the latency combination for streamed municipal chat; distractors add tape delays, eager-only paths, or overnight batching. Caveat: first-token latency also depends on prompt length and batching policy—engine plus cache help, but oversized system prompts or cold starts can still dominate unless warmed. Operational check: measure TTFT and inter-token latency for water-outage prompts with TensorRT+KV on versus off, confirm streamed partial answers remain coherent, and gate release on the utility’s resident-facing SLO.