City hall’s older inference GPU runs out of memory on a 70B permit assistant. Which optimization approach best fits the hardware?
Select an answer to reveal the explanation.
Short Explanation
An old city GPU choking on 70B is like a small elevator with a grand piano—trim first, then pack tighter. Structured prune, then INT8 quantize so permits still run.
Full Explanation
Fitting a 70B permit assistant onto an older city-hall inference GPU requires reducing both parameter count and numeric precision. Applying structured pruning before INT8 quantization shrinks structured capacity first, then packs remaining weights into a lower-bit format that NVIDIA GPUs can serve efficiently—an established optimize-then-quantize pattern for constrained municipal hardware. Adding more FP32 copies of the full 70B model on the same GPU increases memory pressure rather than relieving it and cannot make a grand-piano-sized model fit a small elevator. Disabling all sparsity and keeping every weight at full precision refuses both levers that actually shrink footprint. Moving only the tokenizer to INT8 while leaving the 70B weights untouched leaves essentially the entire memory burden intact; tokenizer tables are tiny relative to transformer weights. Exam caveat: structured pruning must be validated on permit-answer quality—aggressive sparsity can harm rare ordinance citations even when INT8 calibration looks fine on average FAQ text. Operational check: record peak VRAM for the FP16/FP32 baseline versus prune-then-INT8 checkpoint on the exact city GPU, plus a civic permit QA metric, and only promote the optimized build if both memory and quality gates pass.