A transit team worries that delay messages might treat some language communities less clearly than others. Which fairness-oriented evaluation approach should they use?
Select an answer to reveal the explanation.
Short Explanation
Fairness testing is like checking that every platform screen is readable—not just the English one by the main stairs. A/B the prompt variants across languages and compare the results so delay messages stay clear for every community.
Full Explanation
A/B testing prompt variants for fairness of multilingual delay messaging and comparing measured outcomes is the fairness-oriented evaluation approach a transit team should use when delay messages might treat some language communities less clearly than others. Design comparable prompt/translation variants, serve them under controlled experiments or offline evaluations across target languages, and measure clarity, actionability, comprehension, or successful rider responses by language community. Systematic comparison reveals disparate quality that a single English prompt would miss.
Shipping one English-only prompt forever and ignoring multilingual outcome gaps hard-codes inequity into rider communications. Translating messages with an untested random dictionary swap at send time is not evaluation and can produce unsafe or incomprehensible notices. Assuming fairness is achieved if a single bilingual staffer likes one sample substitutes anecdote for measured, multi-community outcomes.
Exam caveat: fairness evaluation for GenAI messaging requires structured comparison (A/B or equivalent) across groups/languages with metrics—not English-only defaults or one-off subjective approval.
Operational check: select language communities in scope, define outcome metrics for delay-message clarity/actionability, run A/B or offline evals on prompt variants, review disparity reports before promoting a winner, and retest when SOPs or model providers change.