A team revises their agent's constraint instructions to fix a reported failure where the agent was modifying files outside the target directory. After the revision, the original failure is resolved. However, the evaluation suite now shows three new failures: the agent refuses to follow symlinks that legitimately point into the target directory, fails to process files in nested subdirectories, and incorrectly rejects valid relative paths. What does this outcome indicate?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Adding a lock to the front door that also locks all the windows and the back gate — that's an overly broad constraint. The fix worked for the specific failure, but the instruction was phrased so broadly it blocked legitimate behaviors too. Narrow constraint revisions and mandatory post-revision evaluation suite runs are the safeguards that catch this. The exam tests whether you know that constraint changes have lateral effects and must always be validated before deployment.
Full explanation below image
Full Explanation
This question tests your understanding of constraint revision side effects — a common failure mode in agent tuning where a fix for one problem is scoped too broadly and inadvertently breaks adjacent, legitimate behaviors.
The correct answer is B. The original instruction change to prevent the agent from accessing files outside the target directory was correct in intent but too broad in implementation. A constraint that correctly blocks '/outside/dir/file.txt' may also block '/target/dir/../../other' (relative path traversal), symlinks that resolve inside the target, or deeply nested subdirectory paths if the pattern matching is too aggressive. The lesson is twofold: (1) constraint revisions should be as narrow as possible, targeting the specific prohibited behavior without blocking similar-but-legitimate behaviors; and (2) the full evaluation suite must be run after every revision to catch exactly these lateral side effects before deployment.
Option A is incorrect. The issue is instruction specificity, not model compatibility. Changing the model would not fix an overly broad constraint instruction — it would reproduce the same problem on a different model.
Option C is incorrect and dangerous. The evaluation suite is correctly identifying real regressions — behaviors that worked before and are now broken. Updating the suite to accept the new broken behaviors would permanently encode the regression as expected, eliminating the safety net for future changes.
Option D is incorrect. Removing symlinks, nested subdirectories, and relative paths from scope is an extreme scope reduction that would make the agent significantly less useful. These are standard file system patterns that any robust file-processing agent should support. The correct fix is to narrow the constraint instruction, not eliminate the features it unintentionally breaks.
The key principle: every constraint revision has lateral effects. Narrow scope changes + post-revision full-suite evaluation is the required workflow.