A fire-watch radio hut is short on GPU memory, and a colleague suggests serving the same refined weights at half precision. What kind of choice is that?
Select an answer to reveal the explanation.
Short Explanation
The fire-watch hut is short on GPU memory. Serving the same refined weights at FP16 or BF16, or an official FP8 compile on H100 via TensorRT-LLM, is a serving-time data-type choice. It is not a homemade numeric type, a shorter prompt, or a cooling ticket.
Full Explanation
Lower-precision tensors such as FP16 or BF16, and the official TensorRT-LLM FP8 compile path on H100, are serving-time data-type choices that shrink memory and can raise throughput. The associate does not write a CUDA kernel to invent a numeric type. A shorter prompt does not change the stored weight format. Precision at serve time is an application config, not a datacenter cooling ticket.