A team improves their documentation-generation agent's output for a specific type of API function by updating the system prompt. After the update, documentation quality for that function type improves significantly. Two weeks later, the team receives bug reports that the agent has stopped generating parameter descriptions for a different function type that was working correctly before the prompt change. The team never ran evaluation against the second function type after making the prompt change. What process was missing?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Fixing one leak without checking if you created another — that's the regression trap. Any time a prompt changes, it can shift behavior across function types that weren't the target of the fix. A regression evaluation suite runs the full battery against the new prompt before deployment so you know both what got better and what got worse.
Full explanation below image
Full Explanation
Prompt changes are global: a modification to the system prompt affects the model's behavior across all input types that flow through that prompt, not just the specific case the change was targeting. This is why regression evaluation is mandatory after any prompt tuning.
A regression evaluation suite is a predefined set of test inputs covering all major usage patterns and previously-validated behaviors. After any prompt change: 1. Run the full regression suite against the updated prompt 2. Compare results to the baseline (pre-change outputs) for each test case 3. Any test case that was passing before and now fails or degrades is a regression 4. Fix the regression before deploying the change
The regression suite should include representative examples from each function type, edge cases, and known previously-failing cases that were fixed in earlier tuning rounds.
Option A (rely on user bug reports) is the status quo that led to the problem — bug reports came two weeks after deployment. Reactive discovery through user reports is too slow and damages trust.
Option C (separate prompt per function type) is a valid architectural approach for isolating changes, but it adds maintenance overhead and does not eliminate the need for regression testing within each prompt's domain. It is a design consideration, not a substitute for regression evaluation.
Option D (fine-tuning vs. prompt engineering) is a separate architectural choice. Fine-tuned models can also introduce regressions; regression evaluation is required after fine-tuning as well. Switching to fine-tuning does not eliminate the need for the missing process.