A team's agent produces excellent code review output but takes 45 seconds per review, making it too slow for developer workflows. Which evaluation and optimization strategy should be applied?
Select an answer to reveal the explanation.
Short Explanation
Sports cars stuck in traffic aren’t useful despite their speed capability. Before changing anything, profile where the 45 seconds actually goes—it might be one slow tool call, a bloated prompt, or an oversized model for the task. Fix the real bottleneck, not just anything that might help.
Full Explanation
Optimizing agent latency requires profiling before optimizing; a common mistake is to reduce quality arbitrarily rather than targeting actual bottlenecks.
Profiling an agent pipeline: 1. Instrument each step, such as the LLM call, tool call, and context preparation, with timing. 2. Identify which step or steps consume the most time. 3. Apply targeted optimizations based on findings: - For LLM call latency, try a smaller, faster model and evaluate whether quality drops are acceptable. - For tool call latency, parallelize independent tool calls and cache frequently accessed results. - For context size, compress or trim unnecessary context that inflates prompt processing time. - For architecture, break sequential chains into parallel calls where possible.
The profiling-and-targeted-optimization strategy is correct because it preserves quality where possible while addressing actual bottlenecks. Quality impact is measured empirically at each step, not assumed.
The accept-the-latency-as-known-limitation approach is wrong because documenting a 45-second latency as a known limitation ignores that it renders the tool impractical. Developer workflows require real-time or near-real-time feedback to remain in flow.
The replace-with-rule-based-system approach is wrong because rule-based systems cannot match the nuanced reasoning quality of LLM agents for code review. Switching to rules trades away the core value proposition.
The reduce-quality-threshold approach is wrong because reducing quality until the agent runs faster is not a principled optimization strategy. It conflates the result, faster execution, with the method, lower quality, without evidence that the quality reduction is necessary or sufficient.