On A100 nodes, county IT wants a stable tax FAQ model quantized after training and must report quality change versus FP16. What strategy fits?
Select an answer to reveal the explanation.
Short Explanation
After the tax FAQ is already trained, INT8 PTQ is the quick squeeze—but prove it. Measure ROUGE against FP16 so the county knows what quality moved.
Full Explanation
On A100-class nodes, a stable tax FAQ model that is already trained is a natural candidate for post-training INT8 quantization, but counties must quantify quality change versus the FP16 baseline. Applying PTQ to INT8 and measuring ROUGE (or an equivalent civic answer metric) against FP16 satisfies the optimize-and-measure requirement rather than assuming bit-width reduction is free. Skipping measurement and assuming INT8 always matches FP16 on tax answers hides regressions on rebate figures and filing deadlines. Training a new model from scratch in binary weights without calibration is a different, harder regime than PTQ and is not the indicated path for an already-stable FAQ checkpoint. Quantizing only at random layers without any civic evaluation metric produces an unvalidated hybrid precision model that cannot be defended to assessor leadership. Exam caveat: ROUGE is one lexical overlap lens—pair it with exact checks on numeric tax amounts and form IDs so PTQ does not silently alter figures that residents act on. Operational check: run the same gold tax FAQ set through FP16 and INT8, publish ROUGE delta plus numeric/ID exact-match rates, and promote INT8 only if both stay within the county’s published tolerance.