A Meridian Line engineer improving the charter-desk prompt changes the system persona, adds two few-shot examples, and raises the retrieved passage count in one iteration. Scores on the evaluation set improve slightly. Why is this a poor way to iterate on a prompt?

Prompt iteration is an experimental loop: hold the evaluation set fixed, change one factor, measure, and record. Bundling the system persona, few-shot examples, and retrieved passage count into one step produces a score that cannot be decomposed; a harmful change, such as extra passages diluting context, may be hidden by a helpful one. Isolating changes also makes it easier to identify what to keep or revert. Separating prompt work into a different workspace does not explain why this result is uninterpretable: the key is to use a fixed evaluation set consistently, while guarding against leakage or contamination. The claim that persona wording and examples or retrieval counts need separate metrics also misses the point: those mechanisms can be compared with the same end-to-end answer-quality measure, and isolating them helps reveal their effects. Nor should you dismiss every slight improvement; small gains can matter, and whether a delta is meaningful depends partly on evaluation-set size and variability. Exam caveat: one-factor-at-a-time testing clarifies attribution, but interactions between factors may still require later combined tests. Practical check: use MLflow to record each variant, its fixed-set results, and traces, then compare runs before promoting a change.