An AI agent built for an HR department is flagged because it frequently answers questions about company benefits by providing accurate historical data from its knowledge base, but users report the answers do not address what they actually asked. For example, a user asking about dental coverage receives a detailed response about vision benefits instead. Which Azure AI Foundry evaluation metric is most appropriate for diagnosing this agent quality problem?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Relevance is like a heat-seeking missile that only asks one thing: did the answer hit the target the user actually aimed at? The agent here is grounded in real documents — it just keeps firing at the wrong target. That gap between what was asked and what was answered is exactly what the relevance metric is designed to surface.
Full explanation below image
Full Explanation
The relevance metric in Azure AI Foundry evaluates how well the agent's response directly addresses the user's query. It is scored on a scale of 1 to 5, where 5 means the response completely and specifically answers what the user asked, and 1 means the response is entirely off-topic even if technically accurate.
The scenario is a classic relevance failure: the agent produces accurate, well-sourced content — so its groundedness score would be high — but the content does not match the user's intent. This is often caused by poor retrieval — the agent retrieves the wrong document chunks — or by a system prompt that does not guide the agent to stay on-topic with the question.
Option A (groundedness) is incorrect. Groundedness measures whether the claims in the response are supported by source documents. Since the agent is successfully retrieving from its knowledge base and the information is factually accurate, groundedness is not the failing metric here.
Option B (fluency) is incorrect. Fluency evaluates grammatical correctness and readability of the response. The scenario does not indicate any language quality problems — the issue is topical mismatch, not readability.
Option D (coherence) is incorrect. Coherence measures whether the response is logically consistent and well-structured within itself. Again, the scenario points to off-topic answers, not structurally inconsistent ones.
For the AB-620 exam, master the four AI Foundry quality metrics: groundedness (factually supported?), relevance (answers the question?), fluency (well-written?), and coherence (logically structured?). When users say 'it didn't answer my question,' the metric is relevance. When users say 'it made up information,' the metric is groundedness.