City IT wants to continue pretraining a decoder LLM on municipal code text so the model learns local ordinance phrasing before any task-specific prompts. Which training objective should they use?

Continued pretraining of a decoder LLM on municipal code should use next-token causal language modeling because decoder architectures are trained to predict the following token given prior context. Running causal LM on ordinance text teaches local phrasing, section structure, and civic legalese before any task-specific prompts or supervised heads are attached. That objective aligns the model's inductive bias with how it will later generate ordinance-aware completions for city IT workloads. Masked language modeling that hides only random ordinance section numbers targets a bidirectional fill-in-the-blank skill typical of encoder or encoder-decoder MLM setups; it is not the standard continued-pretraining objective for a decoder that must later generate left-to-right. Binary classification of whether each section is currently enforced is a supervised downstream task, not domain language adaptation, and it fails when enforcement labels are incomplete or stale. Contrastive image-caption pretraining on city hall photographs optimizes a vision-language alignment objective unrelated to absorbing municipal code text. Exam caveat: causal continued pretraining improves domain fluency but does not by itself guarantee factual currency of ordinances; retrieval or tools are still needed for 'is this section in force' questions. Operational check: after causal continued pretraining on the code corpus, probe with held-out ordinance snippets for next-token perplexity and qualitative phrasing match, confirm no task head was required for the adaptation goal, and only then attach prompt templates for clerk-facing tasks.