A city library's program assistant is overspending tokens on long prompts and oversized retrieval chunks. Which optimization should the team apply?

When a city library’s program assistant overspends tokens on long prompts and oversized retrieval chunks, the team should tighten system prompts and reduce retrieval chunk sizes. Token cost scales with prompt length and retrieved context. Lowering sampling temperature on system prompts and chunks to cut token usage only makes sampling more deterministic and does not shrink the oversized prompt or retrieval chunks sent every turn—including simple questions like storytime hours. Constraining prompt and chunk size cuts spend while keeping enough grounding for accurate answers. Lowering sampling temperature on system prompts and chunks to cut token usage fails by concept: temperature controls randomness, not prompt or chunk size, so the bloated input tokens that drive cost remain unchanged. Caching frequent prompt embeddings and reusing large chunks to cut token usage fails because it only saves embedding work while still sending the same long prompts and oversized chunks to the generative model every turn, so input tokens stay bloated. Migrating to a smaller embedding model to compress chunks and cut token usage fails because embedding model size affects vectors and retrieval quality, not how many tokens of prompt and chunk text get sent to the generative model each turn—the token bill stays bloated. Exam caveat: token optimization for RAG assistants targets prompt and chunk size first—not lowering sampling temperature on system prompts and chunks, caching prompt embeddings and reusing large chunks, or embedding-model swaps that leave prompt and chunk text size untouched. Operational check: shorten the system prompt to essential voice/rules, tune chunk size and top-k for library content, measure tokens per turn before/after, and spot-check that grounding quality remains acceptable on program FAQs.