A 311 chat window needs the first sentence quickly, while a nightly job must classify twelve thousand work-order lines. How should serving be tuned?
Select an answer to reveal the explanation.
Short Explanation
The 311 window wants the first sentence now. The night job wants twelve thousand lines done. Small or dynamic batches for the chat path, large batches for throughput overnight. Same model, opposite serving goals. Training topology is not that knob.
Full Explanation
Interactive and overnight jobs can share a model and still need opposite serving goals. A 311 clerk cares about time-to-first-token and end-to-end latency, which means small or dynamic batches. A twelve-thousand-line night job cares about throughput, which means large batches. Distributed-training topology is not that live-inference choice.