A transit agency is further adapting an open decoder-only checkpoint on rider-complaint narratives. Engineers debate masked language modeling versus causal language modeling. Which choice is appropriate?
Select an answer to reveal the explanation.
Short Explanation
You can keep the decoder reading complaints left to right, predicting each next token as it would continue a ticket. Causal LM fits this checkpoint; masked fill-in-the-blank training is not required for decoder-only models.
Full Explanation
When you further adapt an open decoder-only checkpoint on rider-complaint narratives, causal language modeling is appropriate because the model is designed for left-to-right next-token prediction. Keeping this objective preserves its attention pattern and helps it learn complaint phrasing, delay vocabulary, and rider tone without imposing a bidirectional mask regime. Preferring masked LM on the claim that decoder-only models require bidirectional ordinance masks confuses encoder-style masked language modeling with decoder training; decoder-only stacks typically lack the bidirectional setup that makes masking natural. Disabling all language-modeling loss and training only a regression head on delay minutes abandons language adaptation and turns the task into numeric delay prediction, without learning complaint style. Replacing complaint text with GTFS schedule tables and training a vision encoder swaps modalities and leaves the text decoder unadapted. Exam caveat: span extraction may call for a separate head or an encoder, but that is not a reason to replace causal LM when adapting the decoder checkpoint. Practical check: confirm the configuration uses autoregressive next-token loss on complaint text, then spot-check generated continuations for coherent left-to-right phrasing; reject a pipeline that silently substitutes masked-LM or vision objectives.