A transit FAQ bot’s underlying Bedrock model version is scheduled to change next week. What should the team do to protect rider-facing answer reliability?
Select an answer to reveal the explanation.
Short Explanation
When your transit FAQ bot’s FM version changes, regression-test critical prompt templates against the new version and compare them with golden civic answers before cutover. Rewriting prompts only after complaints, assuming newer versions cannot change behavior, or adding throughput won’t catch prompt-related answer regressions.
Full Explanation
Bedrock model version changes can alter how identical prompt templates behave: tone, instruction following, and refusal patterns may shift even when the prompt text is unchanged. To protect rider-facing reliability, run regression tests of critical prompt templates against the new FM version before cutover and compare the outputs with a golden civic answer set. This makes failures visible before riders encounter them. Deleting all prompts and rewriting them from scratch after riders complain is reactive incident response, not a controlled cutover strategy. Assuming newer FM versions never alter prompt behavior is empirically false, so skipping tests leaves public answers exposed to avoidable regressions. Increasing provisioned throughput addresses capacity and concurrency, not behavioral changes in answers; extra capacity cannot validate or repair prompt behavior. Exam caveat: capacity levers are not substitutes for prompt QA when the issue is a model version change—test behavior against golden civic answers first. Practical check: maintain a golden set of rider FAQ prompts and expected answer properties, run automated comparisons using the candidate model ID, gate the production alias switch on pass criteria, and keep a fast rollback to the prior model version if post-cutover monitors spike.