School-district IEP notes must run on small edge inference boxes in nurse offices. How should the tokenizer vocabulary be sized?
Select an answer to reveal the explanation.
Short Explanation
Nurse-office boxes are not data-center GPUs. Shrink the tokenizer vocab so the IEP model’s embeddings actually fit the edge hardware.
Full Explanation
School-district IEP note models that must run on small edge inference boxes in nurse offices are memory-bound by embedding tables and residual weights. Shrinking the tokenizer vocabulary reduces embedding parameters so the packaged model fits RAM/VRAM limits while still covering the clinical and educational terms needed on-device. The design goal is fit-to-edge, not data-center parity. Growing the vocab until embedding tables alone exceed the edge box RAM directly violates the deployment constraint and guarantees OOM or swap thrashing in nurse offices. Copying a trillion-token web vocab unchanged onto those devices imports irrelevant capacity and embedding cost that edge boxes cannot host. Disabling the tokenizer and feeding raw UTF-16 bytes to the model is not a practical LLM serving pattern for IEP assistants—it explodes sequence length, breaks standard subword modeling, and still fails to address structured vocabulary sizing under edge limits. Exam caveat: vocab shrinkage can raise fertility on rare medical terms; compensate with domain merges for high-value IEP phrases while still keeping total embedding size inside the device budget. Operational check: measure peak RSS/VRAM of the full inference graph including embeddings on the target nurse-office box, and ship only if cold-start load and a timed IEP summarization job succeed without paging.