Evaluating and deploying agentic workflows

GCP-PAA · 44 questions

• A county permitting-office team is assembling a golden dataset for their permit-application chatbot. Most prompts are straightforward status-check questions, but the team adds a smaller set of deliberately ambiguous prompts asking about multiple permit types at once. What makes those ambiguous multi-permit prompts a valuable addition to the golden dataset rather than a redundant variation on the status-check prompts?
• A city 311 team drafts prompts for their golden test set that intentionally include multi-intent resident requests, such as reporting a pothole while also asking about the trash pickup schedule in the same message. What does including these multi-intent prompts stress-test that single-intent prompts cannot?
• A transit authority runs a continuous evaluation pipeline after every release of its trip-planning agent. For each golden scenario, the pipeline checks whether the agent's tool calls, such as fetching the schedule and computing a transfer, match the success criteria defined for that scenario. What is this pipeline verifying that a check of only the agent's final answer text would miss?
• A county IT architect is scoring a courts-clerk agent's document-retrieval accuracy and is choosing among ADK's evaluation tooling (evalset), the Agent Platform Gen AI evaluation service, and a custom autorater. The retrieval logic is unusually domain-specific, hinging on court-filing conventions that generic quality scorers don't recognize. Which choice best fits this requirement?
• A housing authority builds a custom autorater to judge whether an eligibility-screening agent's explanations are policy-compliant, after finding that a generic quality scorer kept rating clearly non-compliant explanations as high quality. What justifies choosing a custom autorater over the generic scorer here?
• A public-health records team is deciding how to evaluate their records-lookup agent. One option scores only the final response returned to the requester; another inspects the full reasoning trajectory, including which tools the agent called and in what order. When is trajectory evaluation the more appropriate choice over final-response-only scoring?
• An open-data portal team evaluates a RAG-backed civic-data assistant against a golden dataset using ADK, scoring retrieved document relevance and generated-answer accuracy as two separate metrics rather than a single combined score. Why does scoring these two dimensions separately give the team a clearer picture than one blended score?
• A water-utility ops team runs their leak-report triage agent's golden dataset through ADK evaluation tooling before every merge, comparing the new scores against a stored baseline from the last known-good build. What does comparing against a stored baseline give the team that a one-off evaluation run does not?
• Before promoting a new build of a tutoring-support agent, a school-district IT team has staff review only the transcripts the pipeline flags as low-confidence, approving or correcting those before rollout rather than reading every transcript. What is the advantage of targeting review at low-confidence transcripts instead of reviewing the full transcript set?
• A clerk's office evaluates its document-classification agent with two distinct metrics: precision and recall for the retrieval step, and task-completion rate for the full end-to-end trajectory. Why does matching each metric to its own pipeline stage matter here rather than reporting one combined completion rate for everything?
• After tightening the system instructions on a permitting chatbot, a city team reruns the full golden-dataset evaluation suite rather than spot-checking a handful of prompts most related to the change. What risk does the full rerun cover that spot-checking the related prompts would not?
• A transit agency compares two candidate versions of its trip-planning agent side by side on the same golden dataset before deciding which to promote to production. What does this comparative evaluation give the team that evaluating each candidate against its own separate baseline would not reliably give?
• A courts-clerk scheduling agent handles multi-turn conversations with self-represented litigants. Its evaluation suite scores not just each turn's individual accuracy but also whether the agent maintains correct case context across the whole conversation. What failure would per-turn accuracy scoring alone miss that this added check catches?
• An emergency-management agent's evaluation suite checks whether it invokes the correct alerting API with correctly formatted parameters, as a step separate from scoring whether its natural-language summary reads well. Why does the team keep these two checks separate rather than folding tool-invocation correctness into the general quality score?
• A housing authority runs its eligibility-screening agent's evaluation suite across demographic subgroups of applicants before deployment, rather than checking only an aggregate accuracy score across all applicants combined. What can subgroup evaluation surface that an aggregate score alone would hide?
• A county IT team wires their ADK agent's golden-dataset evaluation into the CI pipeline so that every code change automatically triggers a full evaluation run, and a merge is blocked if scores drop below a set threshold. What problem does wiring evaluation into CI solve that running the same evaluation manually, only when someone remembers to, does not?
• A library-programs assistant's evaluation scores decline gradually over several months even though no code changed. Investigating, the team suspects the golden dataset itself has gone stale relative to how resident questions have shifted. What does this scenario illustrate about golden datasets that a one-time creation approach misses?
• A 311 assistant's golden dataset deliberately includes out-of-scope resident requests, such as asking the agent to draft a legal complaint, to verify it declines gracefully instead of attempting the task. What does this design choice specifically test that a dataset made only of in-scope requests could not?
• A public-health records RAG agent's evaluation checks whether its cited source passages actually support the generated answer, rather than only whether the final answer text reads as plausible. What distinct problem does grounding evaluation catch that plausibility alone would miss?
• A county evaluation team debates whether their 40-prompt golden dataset is large enough to draw a reliable conclusion before promoting a new permitting-agent version. What consideration should weigh most heavily on whether 40 prompts is enough for this go/no-go decision?
• During development of a transit trip-planning agent meant for a real-time kiosk, the team runs evaluations that jointly track response latency alongside quality scores, since a faster but slightly less accurate model might still be preferred for that use case. Why does the team evaluate latency and quality together rather than optimizing quality alone and treating latency as a separate concern?
• A school-district team periodically compares their custom autorater's scores against a sample independently rated by human staff, to confirm the autorater hasn't drifted from human judgment. What risk does this periodic calibration step address that building the autorater once and trusting it indefinitely would not?
• A city architect is choosing a deployment runtime for a resident-facing 311 chatbot. One team wants managed session state and minimal operational overhead; another wants fine-grained control over the container environment. Which pairing of runtime to requirement fits best?
• A permitting office sees sharp seasonal traffic spikes during building season and needs its agent deployment to scale cost-effectively during quiet months without paying for idle capacity year-round. Which deployment consideration should drive this decision?
• A courts-clerk agent that has reliably answered filing-status questions for months starts giving subtly outdated procedural guidance, even though no code or prompt changes were deployed to it. What is the most likely explanation, and how should the team confirm it?
• A transit agency's trip-planning agent starts responding noticeably slower, and trace spans show most of the added time sits inside a downstream schedule-lookup tool call rather than the model's reasoning step. Where should the team focus its optimization effort?
• A 311 resident-service assistant occasionally gets stuck repeatedly calling the same lookup tool with nearly identical arguments and never produces a final answer for the resident. What production issue is occurring, and what should the team check first?
• A water-utility agent's outage-reporting workflow fails for every resident after a downstream permit-status API it depends on goes offline, rather than only failing the specific requests that call that API. How should the on-call team classify this incident?
• A county IT team wants to instrument their permitting agent so that, when a resident files a complaint about a specific answer, engineers can reconstruct exactly which reasoning steps and tool calls happened for that request. Which Google Cloud Observability capability should they configure for this?
• A transit kiosk agent's engineers want to pinpoint exactly which stage of a multi-step itinerary-building workflow is contributing the most latency before they decide where to optimize. Which capability should they use?
• A public-health records assistant's production monitoring dashboard flags a rising rate of responses that cite records not present in the underlying data source. What production issue does this pattern indicate, and how does it differ from a latency or cost concern?
• A permitting office's agent traffic triples during the spring building-permit rush, and the team wants the deployment to scale to meet demand automatically rather than requiring someone to provision capacity by hand. Which deployment consideration addresses this need?
• Before fully switching resident traffic over to a new version of the 311 assistant, a city team routes a small percentage of live requests to the new version and compares its outcomes against the current production version before expanding further. What deployment strategy is this?
• A school-district agent's newest release starts producing incorrect attendance-policy answers within hours of launch, and the on-call engineer needs to restore correct behavior for families as quickly as possible. What is the most direct deployment action to take?
• A transit authority stands up a full parallel production environment running its new agent version, validates it there, and then cuts resident-facing traffic over to it all at once rather than upgrading the live environment in place. What deployment approach is this?
• A county IT budget owner reviews per-request token consumption alongside compute spend across a fleet of civic agents to determine where production cost is concentrated. What does examining both dimensions together accomplish that either one alone would miss?
• A courts-clerk team needs its coding-agent-built document-review workflow to run inside a tightly controlled sandbox with custom networking rules that a fully managed runtime doesn't expose configuration for. What consideration should drive their deployment runtime choice here?
• A water-utility team is deciding between Cloud Run and Agent Runtime for two different workloads: a stateless outage-lookup agent that answers each request independently, and a stateful multi-turn casework agent for housing-authority intake that must track context across a resident's conversation. How should the workload characteristics guide the choice?
• A city sets alerting thresholds on its Observability dashboards so that a sustained rise in tool-call error rate or response latency for the permitting agent automatically pages the on-call engineer. What production-monitoring practice does this represent?
• An emergency-management agency defines a maximum acceptable response-time target for its flood-warning alerting agent, since a slow response during an active flood warning carries real consequences that a routine 311 inquiry does not. What is this response-time target an example of, and why does it differ across the agency's agents?
• A transit authority deploys its trip-planning agent across two separate regions so that an outage in one region does not take resident-facing trip planning offline entirely. What deployment consideration does this reflect?
• After a resident complaint about an incorrect permit-status answer, a city engineer cross-references Cloud Trace spans with Cloud Logging entries to reconstruct exactly what happened during that single request. What does combining the two sources provide that either alone would not?
• A county IT team uses latency figures observed during development-time evaluation runs to set realistic capacity and autoscaling targets before launching their permitting agent into production. What does this practice connect together?
• A library-programs agent's production monitoring flags a recurring pattern of low-quality answers, and the team adds those flagged transcripts to the golden dataset used for evaluation before redeploying an updated version. What practice does this represent?