Public-health continued pretraining shuffles clinic-note documents across the batch but must still train a decoder with causal language modeling. How should token order be handled?

For public-health continued pretraining that shuffles clinic-note documents across the batch while still using causal language modeling, the correct token-order policy is to shuffle which notes appear in a batch while preserving left-to-right causal prediction within each note. Document-level mixing improves batch diversity and reduces overfitting to filing order, but inside a note the clinical narrative must remain chronological so next-token targets stay meaningful. Causal LM depends on that intact prefix-to-next-token relationship. Reversing every note so the model predicts earlier clinical tokens from later ones inverts causality and trains an unnatural right-to-left skill the deployed decoder will not use for ordinary generation. Randomly permuting tokens inside each note destroys local syntax and medical narrative structure, turning regularization into noise that harms rather than helps clinic-note modeling. Masking the middle half of each note and predicting only masked spans switches the objective toward span MLM and away from full causal next-token training the decoder continued-pretraining plan requires. Exam caveat: shuffle documents, not tokens; if notes are packed into long contexts, still respect causal order within each packed document and use clear separators so boundaries remain recoverable. Operational check: log a few packed batches to confirm note IDs are shuffled across steps while token sequences inside each note stay in original order, and assert the loss remains standard causal LM rather than reverse or masked variants.