Parks chat keeps breaching latency SLOs. Ops wants production automation before a full model retrain. What fits?
Select an answer to reveal the explanation.
Short Explanation
You can address parks chat’s latency without waiting for a full retrain: let automation adjust temperature and max tokens when the latency SLO is breached. Think of it as a thermostat, not a building rebuild.
Full Explanation
When parks chat keeps breaching latency SLOs and ops wants production automation before a full model retrain, automating retuning of temperature and max tokens on SLO breaches is the fitting control. Serving parameters affect decode length and runtime: lowering max tokens directly caps decode length and so cuts latency, while temperature and sampling changes mainly affect output style and only indirectly affect answer length and restore p95 latency without waiting months for a new fine-tune. Tying automation to SLO breaches makes it reactive, bounded, and reversible, like a thermostat, while retraining remains a heavier lever. Waiting months with no changes accepts a degraded resident experience and burns the error budget. Deleting the latency SLO hides the problem rather than restoring parks chat performance. Manually guessing new max tokens once per year without telemetry is too slow and disconnected from actual breach timing. Exam caveat: automated retuning needs parameter bounds, quality checks, and rollback so latency fixes do not silently harm answer usefulness; parameter tuning cannot solve a model or retrieval path that is fundamentally too slow. Practical check: alert on parks latency SLO breaches, apply bounded adjustments to temperature and max tokens, remeasure latency and a short quality sample, and escalate to capacity changes or retraining if breaches persist within safe parameter ranges.