311 supervisors labeled preferred versus rejected triage replies and lack full RLHF infrastructure. Which alignment approach fits?
Select an answer to reveal the explanation.
Short Explanation
When 311 bosses already marked good triage replies versus duds, you don’t need a giant RLHF circus. DPO lines those preference pairs up and nudges the model toward the winners.
Full Explanation
311 supervisors already labeled preferred versus rejected triage replies, which is exactly the pairwise preference signal direct preference optimization consumes. Running DPO on those pairs aligns the municipal triage assistant toward supervisor-approved intents without standing up full RLHF reward-model and PPO infrastructure the city lacks. That approach fits labeled preference data under constrained alignment ops. Launching multi-agent reinforcement learning across every city department before any preference data use invents sprawling complexity unrelated to the labeled pairs on hand. Ignoring preference labels and only maximizing next-token likelihood on rejected replies trains the model to imitate bad triage. Retraining the entire base LLM from random weights using unlabeled 311 audio only discards both the base model and the supervisor labels. On the exam, DPO is the lightweight preference fit when civic preferred/rejected pairs exist without full RLHF; distractors escalate to multi-agent RL, train on rejects, or retrain from scratch on unlabeled audio. Caveat: DPO still needs careful pair quality—noisy or contradictory supervisor labels can encode inconsistent triage policy, so adjudicate disagreements before training. Operational check: build preferred/rejected triage pairs from 311 review, run DPO from the SFT (or base) checkpoint, and A/B win-rate against held-out supervisor preferences before routing live resident tickets.