Transit must meet a p95 latency SLA for rider queries while still using dynamic batching. What should ops tune?
Select an answer to reveal the explanation.
Short Explanation
Dynamic batching is a carpool—great until the van waits forever for one more rider. Cap max and preferred batch sizes so p95 stays honest and lone rider queries don’t starve at the curb.
Full Explanation
Transit rider-query serving that must meet a p95 latency SLA while still using dynamic batching depends on configuring max and preferred batch sizes so the batcher improves GPU efficiency without waiting too long or starving singles. Preferred size targets efficient kernels; max size caps how large a batch can grow; together with queue delay limits they bound wait time for lone rider queries. Setting an infinite max batch and ignoring preferred size lets batches grow without bound and blows p95 during uneven arrival patterns. Disabling dynamic batching so every query times out independently abandons the throughput tool transit needs under load. Preferring only the absolute maximum batch even when single queries starve optimizes utilization metrics at the expense of rider-facing latency. Exam caveat: optimal batch knobs shift with sequence length mix—rush-hour short queries tolerate different settings than long itinerary prompts, so one static max may need daypart or model-specific profiles. Operational check: sweep preferred and max batch plus max queue delay under a recorded rider trace, plot p95 versus tokens per second, and lock the operating point that meets the SLA with the highest sustainable throughput.