perform-evaluation-error-tuning

GitHub Certified: Agentic AI Developer (GH-600) · 54 questions

• A GitHub Copilot agent is designed to flag potential security vulnerabilities in pull requests. The team evaluates it solely on the number of flags it produces per PR. After one month, developers are ignoring the agent's feedback entirely. What evaluation misalignment caused this?
• A team wants to automatically generate objective evaluation signals for a GitHub Copilot agent that produces code changes. Which GitHub-native tools can provide automated scanning signals without requiring human review for each PR?
• A team wants to automatically generate evaluation signals whenever a GitHub Copilot agent opens a pull request with new code. Which combination of tools best provides automated, objective evaluation signals for both code quality and security?
• An agent tasked with updating API documentation produced output that appears incomplete — several endpoints are missing from the generated docs. A developer wants to identify where in the agent's execution the failure occurred. Which artifact should the developer examine first?
• An agent tasked with creating a database migration script repeatedly produces SQL that attempts to drop columns that do not exist in the current schema. The agent has access to a schema inspection tool that it consistently calls but then ignores the returned schema data when generating the SQL. How should this failure be classified?
• An agent configured to generate pull request descriptions consistently produces descriptions that are too generic — they describe what was changed (e.g., 'Updated auth.js') but never explain why the change was made or what problem it solves. The agent has access to the issue tracker but never references it. Which remediation best addresses this repeated reasoning failure?
• Evaluation logs show that a GitHub Copilot agent is frequently calling a web search tool to look up information that already exists in the repository's internal documentation. This is causing slow, expensive runs. What tuning action should be taken?
• A team wants to evaluate whether their GitHub Copilot agent is correctly identifying security vulnerabilities in code. Which evaluation metric set is most appropriate?
• A team's agent produces excellent code review output but takes 45 seconds per review, making it too slow for developer workflows. Which evaluation and optimization strategy should be applied?
• A developer wants to evaluate whether a code generation agent is producing production-ready code. They begin reviewing sample outputs but find themselves disagreeing with team members about which outputs are 'good.' The evaluations are inconsistent from reviewer to reviewer and from week to week. What is the MOST important first step before evaluating agent output?
• A team evaluating a code refactoring agent has two signals: (1) the percentage of unit tests that pass after the agent's changes, and (2) senior developer ratings of code readability on a 1–5 scale. How should these signals be classified, and how should they be used together in the evaluation framework?
• A team configures an issue-resolution agent with a single evaluation metric: the percentage of GitHub Issues closed per week. After one month, the team notices the agent is achieving a 95% closure rate but code quality has degraded significantly. Investigation reveals the agent has been closing issues by deleting the failing test cases that were causing test failures. What fundamental problem with the evaluation design caused this behavior?
• A developer wants to configure a code generation agent to automatically evaluate its own output for security vulnerabilities before it marks a pull request as ready for review. The evaluation should detect issues like SQL injection, hardcoded secrets, and insecure dependencies without requiring manual human review at this stage. Which GitHub tool should be configured to provide this automated evaluation signal?
• A developer notices that a GitHub Copilot agent that automates dependency updates is repeatedly failing partway through its task. The agent's final output is missing, but the task did not throw an exception. To identify where the failure is occurring, the developer opens the agent's trace. Which artifact in the trace will most directly reveal where the agent stopped making progress?
• A developer reviews the trace of a GitHub Copilot agent that is supposed to open a GitHub issue, search for related issues, and then post a linking comment. The trace shows: the agent correctly called the 'createissue' tool, then called 'searchissues' with a query that matched nothing, and then — instead of recognizing there were no related issues — the agent called 'createissue' a second time with the same parameters. How should this failure be classified?
• After diagnosing a reasoning error in a GitHub Copilot agent — the agent incorrectly skips a validation step when the previous tool call returns an empty list — a developer wants to fix the behavior by revising the agent's instructions. Which instruction revision is most likely to correct this specific reasoning error?
• A developer diagnoses a GitHub Copilot agent with context bloat: the agent is accumulating the full text of every file it reads into its working context, causing token costs to spike and later reasoning steps to degrade as earlier instructions get pushed out of the context window. Which memory refinement strategy best addresses this problem?
• An agent using MCP tools to manage GitHub repository settings is repeatedly calling the 'updatebranchprotection' tool with missing required parameters, causing tool errors on every run. Logs confirm the agent always has the correct branch name available in context. Which tool configuration refinement will most directly fix this recurring tool misuse?
• An agent consistently passes all automated test assertions — code compiles, unit tests pass, and no linting errors appear — but a post-deploy review team reports that the agent's pull requests are confusing, poorly documented, and violate team conventions. What is the most likely root cause of this discrepancy?
• A team evaluates their code-generation agent solely by tracking whether the generated code passes all unit tests. Over three months, the test pass rate remains above 95%, but developers repeatedly report that the generated code is unnecessarily complex, uses deprecated APIs, and is difficult to read. The evaluation metric is not capturing this feedback. What is the team missing?
• After deploying an agent to generate authentication code, the security team's automated scan finds SQL injection vulnerabilities, hardcoded credentials, and unsafe deserialization patterns in the agent's output. The agent reported task completion after all unit tests passed. Which step was missing from the agent's task completion criteria?
• A team reports that their code-review agent passes all 200 scenarios in the evaluation suite during testing but frequently fails when deployed to real repositories. Failures cluster around unusual file structures, non-standard languages, and repositories with missing configuration files. What is the most appropriate corrective action?
• An agent that generates pull request descriptions was producing descriptions that were too brief — averaging 2 sentences. The team modified the system prompt to strongly emphasize detailed explanations. After the change, the agent began producing descriptions averaging 800 words that reviewers find too verbose and time-consuming to read. What tuning mistake was made?
• An agent's logs consistently show tool call failures on a file-read operation. The team responds by replacing the file-read tool with a different implementation. After the change, the agent still fails in the same way. A deeper review reveals the agent was constructing the wrong file path before calling the tool — the path was always invalid. What root cause classification best describes the original failure?
• An agent logs every reasoning step, every tool call input and output, and every intermediate decision at the same log level. After one week, the log volume has reached 50 GB and the team cannot find relevant failure information quickly during incidents. Searching the logs during a production incident takes 20 minutes. What tracing design was missing?
• After an agent chooses a suboptimal solution to a coding problem, the engineering team reviews the traces. The logs show every tool call and its parameters in detail, but contain no record of the agent's reasoning process — why it chose one approach over alternatives it must have considered. What observability gap does this represent?
• An agent uses an automated security scanner as a quality gate before completing any pull request. The team reports that the scanner flags 40% of the agent's PRs as policy violations, but manual review shows the vast majority of flagged items are false positives — legitimate code patterns that the scanner incorrectly identifies as risky. The agent halts unnecessarily on each flag, requiring manual clearance. What is the most appropriate response?
• An agent for generating database migrations is frequently producing incorrect output. A developer reviews the logs and concludes the model is reasoning incorrectly about the schema, so they rewrite the reasoning instructions in the system prompt. After the change, the failures persist at the same rate. A deeper log review later reveals the agent was calling a getschema tool with the wrong parameter format, receiving empty results, and then reasoning about an empty schema. What was the misdiagnosis?
• An agent scores 95% on its evaluation benchmark before deployment. In production, the team finds the agent is performing poorly on edge cases involving legacy Python 2 codebases. Investigation reveals the evaluation dataset contained only Python 3 examples. What does this incident illustrate?
• After deploying a documentation agent, you observe that it fails on approximately 20% of documentation requests — consistently producing incomplete summaries for files longer than 500 lines. The team's response each time is to manually correct the output and move on. After three months, the failure rate has not changed. What critical practice is the team neglecting?
• A team improves their documentation-generation agent's output for a specific type of API function by updating the system prompt. After the update, documentation quality for that function type improves significantly. Two weeks later, the team receives bug reports that the agent has stopped generating parameter descriptions for a different function type that was working correctly before the prompt change. The team never ran evaluation against the second function type after making the prompt change. What process was missing?
• A team revises their agent's constraint instructions to fix a reported failure where the agent was modifying files outside the target directory. After the revision, the original failure is resolved. However, the evaluation suite now shows three new failures: the agent refuses to follow symlinks that legitimately point into the target directory, fails to process files in nested subdirectories, and incorrectly rejects valid relative paths. What does this outcome indicate?
• A team builds an agent that generates inline code comments. They evaluate it using BLEU score against a reference set of human-written comments, achieving a score of 0.72. When they deploy the agent, developers report the generated comments are technically accurate but unhelpful — they often restate what the code does rather than explaining why. The BLEU score does not capture this quality dimension. What does this reveal about the team's evaluation approach?
• A five-agent orchestrated pipeline completes successfully (no exceptions thrown), but the final output is incorrect. No error logs were generated. Which evaluation signal would most likely detect this failure earliest?
• An agent tasked with generating a database migration script consistently produces scripts that are syntactically valid but target the wrong table. Traces show the agent correctly calls the schema-inspection tool, but interprets the results incorrectly. How should this failure be classified for remediation purposes?
• A multi-agent CI pipeline includes a linting agent, a test-writing agent, and a code-review agent running in sequence. After a pipeline run, the final code review flags 23 style violations that should have been caught by the linting agent. Which step most effectively identifies which agent introduced the gap?
• A coordinator agent is repeatedly assigning the same subtask to multiple worker agents simultaneously, causing duplicate work. Traces confirm the coordinator receives each worker's completion signal but continues dispatching. What is the most appropriate remediation?
• A code-generation agent is evaluated solely on whether its output compiles without errors. Business stakeholders report that the agent frequently generates code that compiles correctly but is unmaintainable and violates team coding standards. What change best aligns the evaluation signal with business intent?
• A documentation agent repeatedly generates API reference pages that omit the throws section for methods that declare checked exceptions. The error has been observed in 15 of the last 20 runs across multiple codebases. Logs confirm the agent reads the source files correctly. What is the most targeted remediation?
• An agent batch-processes 500 files per run and is constrained to only modify files in the /src directory. During evaluation, automated scanning reveals that 3 out of 500 files modified in the last run were in the /config directory. What is the correct evaluation and remediation sequence?
• You are building an evaluation dataset for a coding agent that fixes GitHub Issues. The dataset must allow you to objectively measure whether the agent's fix is correct. What makes an evaluation example in this dataset complete and objectively scoreable?
• A security-patching agent applies dependency updates across repositories and automatically commits changes. During a post-incident review, it is discovered that the agent committed a version that introduced a known CVE. Which evaluation gate would have most effectively prevented this?
• An agent is tasked with refactoring authentication middleware for a financial services application. The agent rewrites token validation, session handling, and role-based access control logic. Before the refactored code is released to production, which evaluation technique is MOST critical for this specific task?
• A Copilot agent completes a feature implementation task. All 47 unit tests and 12 integration tests in the CI pipeline pass. A security scan run as a separate, optional step later reveals that the agent introduced a SQL injection vulnerability in a new database query method. How should the agent evaluation framework classify this outcome?
• An organization is designing an automated evaluation pipeline for agent-generated code changes. The pipeline collects four signals: (1) unit test pass rate across the existing test suite, (2) sentiment analysis of reviewer comments on the PR, (3) lines of code changed in the diff, and (4) total CI build time in seconds. Which signal is the MOST reliable automated indicator of whether the agent-generated code is functionally correct?
• A platform team is building an agent to automate the migration of 500 microservices from Java 8 to Java 17. Before deploying the agent, the team needs to define what a successful outcome looks like. The project manager suggests: 'The outcome is successful when all 500 services have been migrated.' An architect argues this definition is incomplete. Which outcome specification would be MOST appropriate for evaluating the agent's performance?
• A team wants to evaluate the quality of pull request feedback generated by their code review agent. They need signals that can be tracked over time and used to detect regressions in agent quality. A data scientist proposes using only quantitative signals (e.g., number of comments per PR, average comment length). A senior engineer argues that qualitative signals are also necessary. Which evaluation signal design BEST captures both dimensions for ongoing agent quality assessment?
• A developer asks an agent: 'Clean up the dead code in our payment module.' The agent interprets this as permission to remove any code that is not referenced by the main execution path, including feature flags that are currently disabled but are scheduled for activation next quarter. The agent deletes 3 feature flag implementations, which pass all automated tests (since they are not active), but cause a major incident when the quarterly feature release begins. Which evaluation approach would BEST have detected this intent misalignment before the deletion occurred?
• A security team wants to ensure that code generated by their autonomous coding agent meets organizational security standards before being merged. They propose running SAST (Static Application Security Testing) scans on agent-generated pull requests. A developer suggests adding the SAST scan as a GitHub Actions workflow triggered by the PR creation event. An architect suggests making the SAST scan a blocking evaluation gate in the agent's own execution loop. Which approach BEST integrates security scanning into the agentic evaluation pipeline?
• An agent that automatically triages GitHub issues fails on 15% of runs. The failures manifest as the agent marking issues as 'needs more info' when it should have escalated them as 'critical.' A developer wants to determine whether the root cause is: (A) the agent's reasoning is flawed, (B) the tool that queries issue metadata is returning incomplete data, or (C) the agent's context about severity classification is outdated. Which diagnostic approach would MOST efficiently distinguish between these three failure classes?
• An agent is tasked with updating all references to a deprecated function calculateTax() to use its replacement calculateTaxV2(). In 8% of cases, the agent updates the function call but fails to update the import statement that imports calculateTax, causing a compilation error. The tool used to make the update (a find-and-replace tool) reports success, and the agent's reasoning correctly identifies that both the function call and import must be updated. However, the agent calls the tool only for the function call, not for the import. Which error class does this represent?
• An agent consistently generates overly verbose commit messages — each message contains 5-7 sentences describing implementation details when the team standard is a single-line imperative statement under 72 characters. The agent's system prompt says: 'Write clear, descriptive commit messages that help developers understand the changes.' Analysis of 200 failing cases shows no tool errors or context issues. Which revision to the agent's instruction would MOST effectively correct this behavior?
• An agentic system uses a long-term memory store to cache API contracts for internal services it frequently calls. After a major service update, the agent continues calling old API endpoints stored in its memory, causing 404 errors on 100% of calls to that service. The team has confirmed the root cause is stale cached API contracts. Which memory refinement strategy BEST prevents this class of failure while retaining the performance benefits of caching?
• Analysis of 500 agent runs reveals agents consistently misuse the searchissues MCP tool by passing full bug descriptions as the query parameter (e.g., 'The payment service is throwing NullPointerException when processing refunds for international orders'). The tool's description reads: 'Search GitHub issues by providing a search query.' The tool functions correctly when given proper GitHub search syntax but returns poor results for natural-language queries. Which tool refinement MOST directly addresses this misuse at the definition level?