A team wants to evaluate whether their GitHub Copilot agent is correctly identifying security vulnerabilities in code. Which evaluation metric set is most appropriate?
Select an answer to reveal the explanation.
Short Explanation and Infographic
A smoke detector that always fires is 100% recall but useless precision — you stop trusting it. One that never fires is 100% precision on its (zero) alerts but misses every real fire. Security scanning evaluation needs both: precision tells you how often the alarm is real, recall tells you how many real fires you're catching.
Full explanation below image
Full Explanation
Evaluating a security vulnerability detection agent requires metrics that capture both the rate of correct identifications and the rate of missed issues:
- Precision = True Positives / (True Positives + False Positives) — of all the vulnerabilities the agent flagged, what fraction were real? - Recall = True Positives / (True Positives + False Negatives) — of all the real vulnerabilities in the dataset, what fraction did the agent find? - F1 Score = Harmonic mean of precision and recall — a single balanced metric.
These metrics require a labeled ground-truth dataset: a set of code samples with known vulnerability locations.
Why B is correct: Precision, recall, and F1 are the standard information retrieval metrics for classification tasks like vulnerability detection. They capture the quality of the agent's judgments, not just its activity.
Why A is wrong: Response time and token count measure efficiency, not detection quality. A fast agent that misses 80% of vulnerabilities is a poor security scanner regardless of its speed.
Why C is wrong: Developer satisfaction ratings are subjective and do not capture objective detection quality. A developer might rate an agent highly even if it misses critical vulnerabilities they were unaware of.
Why D is wrong: Counting flags per file rewards verbosity, not accuracy. An agent that flags every function in every file would score high on this metric while producing useless noise.