After a tram booth adds “car barn” to Riva ASR, a supervisor wants a number before keeping the change. How should the team measure whether the customization helped?

After a tram booth adds “car barn” to Riva ASR, the team should measure whether customization helped by comparing ASR variants with a word-level error style metric on the same held-out utterances. Civic speech trials need a number the public can trust: frozen evaluation audio and comparable error counts, not hallway vibes. That measurement discipline protects municipal deployments from shipping feel-good vocabulary changes that never reduced transcription mistakes riders depend on for wayfinding.

Asking whether it sounded better in the hallway with no fixed utterances fails by replacing controlled evaluation with anecdotal listening that cannot be repeated. Comparing only loudspeaker loudness fails by measuring amplification hardware instead of recognition accuracy on domain phrases. Scoring only by how colorful the wayfinding map looks fails by importing an unrelated visual aesthetic as if it were a speech metric.

Exam caveat: ASR customization success is word-level error on held-out speech, not loudness or map art, and informal “sounds better” is not an associate-grade result. Operational check: freeze a labeled utterance set that includes “car barn” and peer depot phrases, decode with stock versus customized Riva under identical audio, and keep the change only if word-level error improves on that sealed set.