A team wants to evaluate whether their GitHub Copilot agent is correctly identifying security vulnerabilities in code. Which evaluation metric set is most appropriate?
Select an answer to reveal the explanation.
Short Explanation
A smoke detector that always fires has great recall but no precision, and one that never fires has precision on nothing but misses every fire. You need precision, recall, and F1 against labeled ground truth to measure both real alerts and missed vulnerabilities.
Full Explanation
Evaluating a security vulnerability detection agent requires metrics that capture both the rate of correct identifications and the rate of missed issues:
- Precision = True Positives / (True Positives + False Positives) — of all the vulnerabilities the agent flagged, what fraction were real? - Recall = True Positives / (True Positives + False Negatives) — of all the real vulnerabilities in the dataset, what fraction did the agent find? - F1 Score = Harmonic mean of precision and recall — a single balanced metric.
These metrics require a labeled ground-truth dataset: a set of code samples with known vulnerability locations.
Why the precision/recall/F1 set is correct: Precision, recall, and F1 are the standard information retrieval metrics for classification tasks like vulnerability detection. They capture the quality of the agent's judgments, not just its activity.
Why the response-time and token-count metric set is wrong: Response time and token count measure efficiency, not detection quality. A fast agent that misses 80% of vulnerabilities is a poor security scanner regardless of its speed.
Why the developer satisfaction scale is wrong: Developer satisfaction ratings are subjective and do not capture objective detection quality. A developer might rate an agent highly even if it misses critical vulnerabilities they were unaware of.
Why the flags-per-file count is wrong: Counting flags per file rewards verbosity, not accuracy. An agent that flags every function in every file would score high on this metric while producing useless noise.