A school district's multi-model endpoint hosts dozens of per-school models, and the team notices frequent model swapping and latency spikes because the endpoint is undersized. How should the team right-size the endpoint's instance type?
Select an answer to reveal the explanation.
Short Explanation
A multi-model endpoint is like a shared filing cabinet for dozens of schools — if the cabinet only has room for a few folders at a time, staff keep swapping folders in and out and everyone waits. Sizing the instance to the combined footprint of the models actually being used together, not just one folder's size, is what stops the swapping and the latency spikes that come with it.
Full Explanation
SageMaker multi-model endpoints load and unload models dynamically from a shared instance's memory as requests come in, so when the combined size of the models in active rotation exceeds available memory, the endpoint has to repeatedly evict and reload models — that repeated swapping is exactly the latency spike the district is seeing. Sizing to just the single largest per-school model ignores that several schools' models may need to be resident simultaneously during a busy period, so it under-provisions the same way the district's current setup already does. Using headcount as a sizing proxy substitutes an unrelated number for the thing that actually matters, model memory footprint, and provides no real guarantee about capacity. Simply trimming the number of hosted models without reassessing instance sizing might reduce swapping, but it does so by cutting capability rather than fixing the underlying mismatch between aggregate model size and available memory — a workaround, not a right-sizing decision. Scope caveat: usage patterns can shift over a school year, so aggregate footprint should be re-evaluated periodically rather than sized once and left alone. Operational check: monitor the endpoint's model-loading and eviction metrics after resizing to confirm swapping drops to an acceptable rate during peak concurrent access.