A transit agency compares two candidate versions of its trip-planning agent side by side on the same golden dataset before deciding which to promote to production. What does this comparative evaluation give the team that evaluating each candidate against its own separate baseline would not reliably give?

Side-by-side comparative evaluation controls for confounding variables by running both candidates against the exact same prompts under the same conditions, so any score difference can be attributed to the agents themselves rather than to drift in the dataset, scoring rubric, or evaluation environment between two separate runs taken at different times. Evaluating each candidate against its own historical baseline risks exactly that confound, since baselines captured at different points may reflect a dataset that has since been revised or an environment that has since changed. Neither approach can promise zero production incidents; evaluation reduces risk, it doesn't eliminate it, since production traffic always includes cases outside the golden dataset's coverage. Running two evaluations does not inherently cost less compute than running one — comparative evaluation typically costs about as much as two separate runs, so speed isn't the justification. Model-version confirmation is an infrastructure check unrelated to what a side-by-side quality comparison demonstrates. Scope caveat: a comparative win on the golden dataset still needs to be paired with monitoring after rollout, since the dataset can't anticipate every production scenario. Operational check: confirm both candidates were scored using the identical prompt set and rubric version before trusting the comparison's result.