A Databricks Model Serving endpoint has P99 latency of 500ms but the SLA requires P99 < 100ms. Which optimization should be tried first?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Latency reduction usually means more compute (bigger instance, GPU) or a leaner model (quantization, pruning, distillation). REST API overhead is typically <5ms and isn't the bottleneck.
Full explanation below image
Full Explanation
Databricks Model Serving latency optimization approaches (in order of impact): 1) Compute size — upgrade from Small (1 CPU core) to Medium or Large, or enable GPU serving for neural networks. Each size increase can halve latency. 2) Model optimization — quantization (FP32 → FP16 or INT8) reduces memory and computation. Model pruning removes unnecessary weights. Knowledge distillation trains a smaller model. ONNX conversion for optimized runtime. 3) Feature retrieval — if online feature store lookups add latency, optimize the online store configuration. 4) Warm-up — Model Serving scales to zero when idle (cold start latency). Enable 'scale-to-zero disabled' or provision minimum replicas. 5) Caching — for repeated identical requests, application-level caching can bypass the model. Increasing model complexity would INCREASE latency. Switching to batch for all requests doesn't satisfy real-time SLA requirements. MLflow logging doesn't run during inference.