A team wants to automatically generate evaluation signals whenever a GitHub Copilot agent opens a pull request with new code. Which combination of tools best provides automated, objective evaluation signals for both code quality and security?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Automated scanning is like installing smoke detectors instead of paying someone to smell for smoke 24/7. GitHub Actions runs on every push, linters catch style and logic issues, and CodeQL finds security vulnerabilities — all without human intervention. Wiring these as required status checks means the agent's PR can't merge until the signals are green. That's the automated evaluation pipeline the exam is looking for.
Full explanation below image
Full Explanation
Automated scanning tools are a cornerstone of generating repeatable, scalable evaluation signals for agent-produced code. The key characteristics of a good automated evaluation pipeline are: it runs without human intervention, it produces objective and measurable outputs, and it gates merges on the results.
Option B is correct. GitHub Actions can be configured to trigger on pull request events, including those opened by agents. Running a linter (such as ESLint, flake8, or Rubocop) produces a code quality signal — style compliance, potential logic errors, complexity warnings. Running CodeQL produces a security signal — static analysis for known vulnerability patterns like SQL injection, cross-site scripting, and unsafe deserialization. Surfacing these as required status checks means a pull request cannot be merged unless both pass, creating a hard gate on agent output quality.
Option A (agent self-reporting) is unreliable. Agents can generate plausible but inaccurate self-assessments. Self-reporting is not an objective evaluation signal — it is subject to the same reasoning errors that may have produced the flawed code in the first place.
Option C (rotating human reviewer) produces qualitative signals and introduces delay, inconsistency, and bottleneck. It does not scale with high-frequency agent activity and is not automated.
Option D (comparing to a reference solution) is a valid technique in some contexts (such as benchmark evaluation), but it is fragile for real-world agent tasks — there is rarely a single correct reference solution, and branch-based comparison does not generalize across diverse agent tasks. It also does not cover security scanning.
In practice, an automated evaluation pipeline for agent PRs typically combines: linters, static analysis (CodeQL, Semgrep), test runners with coverage reporting, and dependency vulnerability scanners — all surfaced as required GitHub Actions status checks.