A town-band festival intern loads model weights from checkpoint A and a tokenizer from checkpoint B because both listings look similar. Token IDs come back garbled. What framework rule was broken?
Select an answer to reveal the explanation.
Short Explanation
Weights from checkpoint A and a tokenizer from checkpoint B look similar and still garble the IDs. Pair the tokenizer and the model from the same hub revision so the vocabulary matches the embeddings. An NCCL gather or a cropper will not fix a mismatched vocab.
Full Explanation
Tokenizer vocabulary and model embeddings are a matched pair, so both must come from the same hub revision. Mixing two similar-looking checkpoints maps IDs the model never learned. An NCCL gather does not repair a vocabulary mismatch. Festival listings here are text checkpoints, not a multimodal crop.