Housing-authority inference latency spikes line up with heavy host↔device copies in the profiler. What is the right response?
Select an answer to reveal the explanation.
Short Explanation
When you see the profiler catching heavy traffic across the PCIe bridge, you’re likely paying for unnecessary trips between host and device. Keep more preprocessing and intermediate tensors on the GPU, and latency can settle as that shuttling stops.
Full Explanation
Housing-authority inference latency spikes that align with heavy host↔device copies in the profiler are classic PCIe thrash: preprocessing or intermediate tensors bounce across the bus instead of staying where the model runs. Keeping more preprocessing and intermediate tensors on GPU reduces those transfers, helping serving latency stabilize under resident load. Copying every token to host RAM after each layer for debugging multiplies transfers and can dominate p95, even if convenient during development. Forcing synchronous pageable transfers on every microbatch prevents pinned asynchronous DMA and maximizes stall time on the bus. Disabling GPU inference and serving only from laptop CPUs at city hall abandons the accelerator path and cannot meet interactive latency targets at scale. Exam caveat: keeping tensors on GPU helps only when device-memory headroom exists; under tight VRAM, selective pinned H2D transfers of compact features may still be required, but bulk per-layer round trips remain the wrong default. Practical check: capture a profiler trace during a spike window, compare host↔device copy time with compute-kernel time, then remeasure p50 and p95 after moving preprocessing and intermediates on-device. Confirm that copy time falls along with the latency spikes.