Public-health agents must be scored on correct triage category and on refusing diagnosis. How should the evaluation pipeline be designed?
Select an answer to reveal the explanation.
Short Explanation
For public-health agents, you need to check both whether they assign the right triage category and whether they refuse to diagnose. A multi-metric evaluation keeps both goals visible; average tokens per reply or friendliness alone can miss a clinical-scope failure.
Full Explanation
Public-health agents must be evaluated on both correct triage category and required refusal to diagnose. A multi-metric pipeline should jointly track triage accuracy and diagnosis refusal, keeping both clinical-scope goals visible. These are distinct ways an agent can fail: it may assign the wrong triage category, or it may give unauthorized medical advice. Using only average tokens per reply does not measure either goal, so an agent could score well on that metric while getting triage wrong or diagnosing. Scoring only friendliness rewards a pleasant tone while ignoring whether the agent stays within clinical scope. Counting every answered symptom question as successful, even when the agent also gives a diagnosis, treats a failure to refuse diagnosis as a success and conflicts with the required boundary. Exam caveat: define a pass criterion for each metric; strong triage accuracy does not make up for a diagnosis when refusal is required. Practical check: evaluate a public-health test set for triage accuracy and diagnosis-refusal performance separately, and fail builds that regress on either measure.