A 311 phone line converts speech to text before an LLM agent triages the call. Where should audio models fit in that multimodal pipeline?
Select an answer to reveal the explanation.
Short Explanation
Phone 311 is a relay: ears first, then the brain. Speech-to-text (or audio understanding) sits upstream and hands a transcript to the LLM agent. Dumping raw voicemail bits into the LLM is the wrong stage order.
Full Explanation
On a 311 phone line, audio must become text (and optionally structured audio understanding) before a typical LLM triage agent can apply routing policies, tools, and prompts designed for language. Placing speech-to-text upstream so transcripts feed the LLM agent is the correct multimodal pipeline fit: audio models handle capture and transcription; the LLM agent handles civic triage over text plus tools. Replacing triage with a raw-audio-only classifier, pasting voicemail binaries into the LLM context, or running the LLM before audio exists all break that staged design.
Replacing the entire triage agent with a single raw-audio classifier and no transcript fails by concept: it abandons flexible language understanding, tool use, and auditable text records cities need for 311. Skipping speech-to-text and pasting voicemail binary into the LLM context window is not a supported or reliable interface for municipal LLM stacks. Running the LLM before audio capture so transcripts are unnecessary reverses causality—there is nothing to triage until speech is acquired and converted.
Exam caveat: STT error rates affect triage quality; confidence scoring, confirmation prompts (“I heard a water leak on Oak Street—is that right?”), and human escalation remain necessary. Store transcripts under the city’s retention and privacy rules.
Operational check: place test calls with known intents, inspect upstream transcripts for adequacy, confirm the LLM agent triages from that text, and verify the pipeline fails closed with a resident-safe message if STT is unavailable.