County IT wants encoder weights that already understand property-record language before classifiers train. What pretraining approach fits?

County IT wants encoder weights that already understand property-record language—parcel jargon, deed clauses, occupancy codes—before classifiers train on vacant-versus-improved or similar labels. Training the encoder with masked language modeling on property-record text builds those civic-aware foundations so downstream heads inherit domain vocabulary and local phrasing instead of starting from generic web English alone. That pretraining approach is the conceptual fit for encoder-first municipal NLP. Training only a decoder with open-ended generation and never masking property tokens targets generative chat, not an encoder backbone for classification. Skipping pretraining and randomly initializing the encoder right before each classifier run throws away transfer and yields unstable, data-hungry civic models. Fine-tuning classifiers exclusively on pixel screenshots of deeds with no text objective ignores the textual nature of property records and forgoes MLM’s language signal. On the exam, MLM on domain text is the encoder pretraining answer for civic-aware weights; distractors use decoder-only generation, random init each run, or screenshot-only paths. Caveat: MLM on deeds can absorb historical bias in property language; pair later classifiers with fairness checks on protected neighborhoods rather than assuming pretraining is neutral. Operational check: after MLM on the property corpus, probe masked parcel/deed sentences for sensible fill-ins, then freeze the encoder and train the classifier—confirm lift versus a non-pretrained encoder on a held-out assessor slice.