Operating, Monitoring, and Securing ML and AI Solutions

MLA-C02 · 48 questions

• A housing authority's SageMaker endpoint scores casework eligibility, and applicant income data shifts after a policy change even though the endpoint keeps returning responses without errors. Staff want to catch this shift before it silently degrades decision quality. Which monitoring approach addresses this?
• A city's 311 resident-service chatbot, built on a Bedrock agent, starts cutting off long answers mid-sentence during peak call volume. The ops team wants an automated way to detect this truncated-streaming pattern rather than relying on resident complaints. Which capability should they use?
• A transit authority's ridership-forecasting SageMaker endpoint has predictions quietly diverging from the baseline established at launch. The team wants an approach that flags this gradual divergence automatically, without manually reviewing every batch of predictions. Which approach fits?
• A permitting-office Bedrock agent calls a Lambda tool to check permit status, and the tool intermittently times out, causing the agent to return incomplete answers to applicants. Which monitoring approach directly surfaces this coordination problem?
• A school district's content-review foundation model is periodically re-evaluated to confirm output quality hasn't degraded as prompts and source documents change over the semester. Which capability fits this recurring re-evaluation need?
• A court clerk's office wants to compare a new document-classification model variant against the current production model on live traffic before fully switching over. Which approach lets them make that comparison safely?
• A water utility's leak-detection anomaly model begins producing an unusual spike in invocation errors overnight, and on-call staff need to be paged automatically rather than discovering the problem the next morning. Which setup achieves this?
• An emergency-management dispatch system uses multiple coordinating Bedrock agents, and one sub-agent silently stops responding to hand-offs during a storm event, delaying triage. Which monitoring approach is needed to catch this kind of failure?
• A county open-data portal's RAG search assistant starts returning less relevant document excerpts to residents after a content refresh, even though the underlying foundation model hasn't changed. Which diagnostic path should the team prioritize?
• A public-health department's case-report text classifier begins seeing incoming reports that look statistically different from its training corpus after a new intake form rolls out. Which monitoring capability is needed to catch this?
• A permitting chatbot team wants a dashboard that surfaces a rising hallucination rate across resident conversations, flagged by automated evaluation rather than manual transcript review. Which combination of capabilities supports this?
• A transit maintenance-prediction model's accuracy is checked periodically against the baseline recorded at deployment time, with alerts triggered when accuracy falls below an agreed threshold. What monitoring practice does this describe?
• A 311 assistant's agent begins truncating streamed responses more often specifically during evening peak load, and staff need to confirm the pattern correlates with load rather than a recent code regression. How should they investigate?
• A housing authority's document-extraction pipeline built on Textract, feeding a downstream model, shows a sudden spike in extraction errors after a new scanner is introduced. Which monitoring setup helps the team catch and localize this kind of issue going forward?
• A school-district assistant team wants to validate a newer foundation model against the currently deployed one before swapping the production model ID. Which approach gives them a reliable basis for that decision?
• A courts docket-summarization agent occasionally fails to retrieve case details from a case-management API tool, and staff need automated alerting when this tool-call failure rate rises. What should the team configure?
• A county deploying a custom document-classification model to a SageMaker endpoint needs to pick an instance family that balances the permitting chatbot's latency needs against monthly compute cost. What should guide this decision?
• A transit authority's steady, always-on forecasting endpoint is currently billed on-demand, and the team wants to know when a longer-term purchasing commitment would actually reduce cost. What should guide that decision?
• A 311 assistant's Bedrock inference bill grows faster than call volume, and the team traces the mismatch to unusually long prompts being sent per resident query. Which monitoring approach would have surfaced this cost driver earlier?
• A housing authority's RAG chatbot recomputes document embeddings on every query, and the team wants a cost-optimization change that doesn't hurt answer freshness. Which change addresses this?
• A county's OpenSearch Service vector index backing an open-data RAG assistant grows steadily each month, and finance asks engineering to explain and control the storage cost trend. What should engineering identify as the underlying cost pattern to monitor and manage?
• A water utility team wants one place to see both latency and spend for its leak-detection inference endpoint, so they can spot a performance regression and a cost spike together rather than checking two separate tools. What should they build?
• A school district's multi-agent tutoring assistant handles many concurrent student sessions, and administrators want visibility into which sessions are consuming disproportionate compute and tokens. Which capability gives them that per-session view?
• An emergency-management department's multi-step triage agent feels sluggish end to end, and engineers need to see exactly which tool call inside the chain is adding the most latency before they change anything. Which approach pinpoints the slow step?
• A permitting office wants to be notified automatically the moment its monthly Bedrock inference spend crosses a set dollar threshold, well before the finance team's monthly report is due. Which approach meets this need?
• A public-health analytics team runs large nightly batch-inference jobs and wants to cut compute spend without changing the model or missing the overnight turnaround window. What should the team do?
• A transit authority discovers several SageMaker endpoints still running at low utilization long after the pilot programs that used them wrapped up, quietly consuming budget every day. What is the appropriate fix?
• A 311 resident-service agent's observability data shows one particular tool integration consistently driving excess token consumption compared to the agent's other tools. How should the team use this signal?
• A county's RDS-with-pgvector knowledge base backing its open-data assistant costs more to query than the team expects at its current configuration, and they want to bring per-query cost down. Which lever should they examine first?
• A housing authority runs predictable nightly batch-transform casework-scoring jobs alongside a spikier interactive agent that resident-facing staff use during business hours. How should the authority choose purchasing options for the two workloads?
• A school district runs a model-evaluation job against every single prompt-template change and notices its evaluation costs climbing faster than the pace of actual model updates. What should the district do?
• A courts clerk office's document-classification endpoint is provisioned for business-hours peak load but sits mostly idle overnight, and the team wants to see this pattern clearly before deciding what to change. What should they do first?
• A permitting chatbot's backend service, running as an AWS Lambda function inside the office's own account, needs to call a Bedrock model. The team is deciding between issuing a Bedrock API key and attaching an IAM role to the function for this access. Which choice best fits least-privilege service-to-service access?
• A housing authority's casework assistant must never surface an applicant's Social Security number in a generated response, even when that number appears in the retrieved case notes the model is grounded on. Which control addresses this directly?
• A court clerk's document-inference SageMaker endpoint handles sensitive docket text and must not be reachable from the public internet. Which deployment choice satisfies this requirement?
• A school district's CI/CD pipeline for model-training code needs an automated check that catches insecure code patterns before the code is deployed. What should the pipeline add?
• A transit authority's container image for a SageMaker inference job must be scanned for known vulnerabilities before it is pushed to the registry used in production. Which capability fits this step?
• A county IT team notices a data scientist's IAM role has access to an entire S3 bucket when the role's job only requires reading a single training-data prefix within that bucket. What is the appropriate remediation?
• A public-health department must be able to show an auditor exactly which staff account invoked a particular Bedrock model call and when, as part of a records-handling review. Which service provides that auditable record?
• An emergency-management department wants continuous detection of any SageMaker endpoint deployed without encryption at rest, across all of its AI workloads, rather than a one-time manual audit. Which approach fits?
• A water utility's SageMaker notebook instance, used to analyze SCADA sensor data, should only be able to reach the specific internal services it needs rather than the broader network. Which combination of controls achieves this?
• A permitting chatbot occasionally drifts into answering unrelated or inappropriate resident questions, and the office wants an automated boundary on the topics and content the agent will respond to, rather than relying on manual review of transcripts. What should the office configure?
• A housing authority discovers a cross-account trust relationship that lets a partner agency's role read its model artifacts more broadly than the partnership actually requires. What is the correct remediation?
• A courts docket-summarization agent needs to let a third-party vendor's external system call it, and the office must decide which credential mechanism fits that integration. Which choice is the better fit?
• A school district's training-data S3 bucket holding student records must be encrypted with a customer-managed key, and only the training role should be able to decrypt objects in that bucket. What should the team configure?
• A county's open-data chatbot has a public-facing endpoint and an internal knowledge-base backend, and the architecture must keep the two properly separated on the network. How should the county lay this out?
• A 311 resident-service assistant program must maintain a defensible record of every configuration change made to its deployed agent, ahead of an upcoming compliance review. Which combination of services builds that auditable trail?
• A transit authority's CI/CD pipeline deploys SageMaker models automatically, and the deployment role currently has access to every model package group in the account instead of just the one it manages. What is the appropriate fix?