ML Solution Monitoring, Maintenance, and Security

MLA-C01 · 72 questions

• A lock-chamber wait endpoint is live. Feature histograms look different from last season, but nobody has labeled true wait yet. Which SageMaker Model Monitor type can run on inputs alone, and which one must wait for outcomes?
• A compost-demand endpoint has been scoring for two weeks. The team creates a Model Monitor schedule and gets empty runs. Capture of request and response payloads was never turned on. What is the prerequisite?
• An ice-rink occupancy model is in production. Staff want alerts when something looks off, but they never computed baseline statistics or constraints. They are comparing live captures to last night's captures only. What should they build first?
• Water-plant turbidity features that were always populated now arrive 40 percent null, and one sensor column flipped from numeric to a status string. Invoke count and latency look fine. Which monitor detects that?
• Kayak-share rebalance scores are still 200s, but dispatchers say predicted needs van is wrong all week. Captured predictions sit in S3; the true van-needed labels land a day later in another prefix and were never joined. What should they run?
• A boat-ramp permit model passed a pre-training Clarify check. After a new marina joined, a protected-group facet's live selection rate left the allowed band. This is a production-bias schedule, not a one-off training report. What should they run?
• A harbor-ticket router still meets its F1 gate, but the SHAP ranking that used to lead with category and hour now leads with a rarely used zip field. Leadership wants an alert on attribution order, not another accuracy job. Which monitor fits?
• Dock-plank photos still look like last year's training set, but what counts as must-replace this week changed after a new ordinance. A data-quality monitor stays green while business outcomes rot. What kind of drift is that?
• Pier-vendor no-show features have shifted after a ferry schedule change. The team already used Clarify at train time to explain coefficients. They now need a recurring check that the live feature and prediction distributions moved. Which capability fits?
• Pier-light-outage scoring runs all night. A one-off notebook comparison was run after a complaint and then forgotten. Official Model Monitor skill includes a repeating schedule. What should they create?
• Compost-plot demand is scored once a week with SageMaker batch transform. There is no real-time endpoint. Staff assume monitoring is an endpoint-only product. Can Model Monitor still run?
• A lock-chamber fog-risk model writes a nested payload the prebuilt monitor cannot constrain. The team does not need a new AWS service; they need custom analysis inside the Model Monitor job. Which container choice fits?
• Ice-rink intake monitors write constraintviolations objects to S3. Nobody looks in the prefix until a Friday incident. Official flow ends in Amazon CloudWatch notifications. What should they add?
• Ferry-depot delay models currently retrain every Monday because that is when the pipeline hook was built. A data-quality monitor has been red for ten days while the calendar job is still five days away. Ops must choose the trigger, not redesign the pipeline. Which trigger fits?
• Two approved bookmobile hold-wait models already share live traffic. The Domain 3 weight job is done. For two weeks ops must compare live business and model metrics on both variants and keep both serving customers, not a shadow copy and not a canary bake. What is that window?
• Pier-meter occupancy variants ran 70 / 30 for fourteen days. Error rate and revenue-per-space are now in CloudWatch. Leadership wants a documented winner before anyone changes weights again. What should ops do?
• Marina inventory invokes suddenly return empty bodies with 200s, and the capture prefix stops growing. Model quality reports are stale, not red. What kind of problem is that?
• A lock-sensor Glue job that builds the hourly feature file started writing zero-row outputs after a schema change. The endpoint is healthy; scores are just defaults on empty input. What should they monitor?
• Pier-attendance hosts sit at 20 percent CPU with clean 200s, so the on-call treats the system as healthy. Ticket-office outcomes have drifted for a month. Official 4.1 knowledge includes ML Lens monitoring principles. What is missing?
• Harbor ticket-risk schedules finish Completed every night. Staff never open the statistics, constraints, or constraintviolations objects, then claim monitoring is on. What should they do?
• A land-records vault hosts twenty rarely used classifiers on one SageMaker multi-model endpoint. Staff want one Model Monitor schedule that reports drift for each classifier. What should they do?
• A licensing desk captures hourly wait-time scores all day. True wait times are keyed the next morning. A model-quality monitor that runs at midnight always fails or scores an empty label set. What should they do?
• A ferry-slot scoring endpoint “feels slow” at the ticket window. Nobody has opened Amazon CloudWatch. What is the first infrastructure-health check?
• After a container regression, a snow-plow crew scoring endpoint started returning 5XX. Model Monitor baselines are still green because few successful payloads were captured. What should operations alarm on?
• Culvert-rating scoring has a 400 ms p99 latency budget. Average ModelLatency looks fine, but p99 stayed over budget for an hour and nobody was paged. What should they configure?
• A tree-canopy pest endpoint fails for one payload shape. Metrics show a small error bump, but the team has no log group and cannot see the exception text. What should they enable?
• Three days of container logs sit in a CloudWatch log group. Operations needs the count of timeout messages by hour, not a manual scroll. What should they use?
• A meter-space wait scorer already runs on AWS Lambda. Cold starts and memory max-out are the complaint, and there is no SageMaker host to chart. What should they use?
• A business-license kiosk calls Amazon API Gateway, then AWS Lambda, then a SageMaker endpoint. Users see about 2-second waits. Each hop’s own chart looks acceptable. What should they use to find which hop owns the latency?
• Trail-crew dispatch opens six separate CloudWatch pages every morning to check utilization, latency, errors, and invocations. What should they build?
• Recreation finance wants a weekly, shareable view of endpoint cost and invoke volume by trail district, joined to a tagging export. A CloudWatch metrics widget is too narrow and dies with the console session. What should they use?
• Someone removed a production variant from a marina scoring endpoint last Tuesday. CloudWatch has host metrics, not who called UpdateEndpoint. What should they create?
• When a clerk-approved model is registered, retraining should start from that event, not from a help-desk ticket. The pipeline already exists. What should they wire?
• A compost-route training job flipped to Failed at 02:00. The page came from someone refreshing the console. How should infrastructure failures notify the team?
• A bus-bunching predictor already meets its model-quality gate. Leadership still asks whether the platform was healthy: how busy the hosts were, how many scores per minute, and whether the endpoint stayed up all weekend. Which KPIs answer that?
• The only real-time host for ice-rink wait scoring died during a patch. InstanceCount was pinned at 1 for cost, so availability hit zero. What operational requirement did they skip?
• A dock-camera slip classifier sits on ml.g4dn.xlarge because “vision uses GPU.” Nobody has benchmarked families or shapes for cost and latency. What should they run?
• An EC2 fleet that runs nightly feature builds for a landfill model is about twice the vCPU it uses. Inference Recommender does not see those boxes. What should they use?
• A wide tabular parcel-appraisal model pages heavily on a compute-optimized host. CPU is not saturated. What should they change?
• An address-line matcher was parked on a GPU instance “in case vision is added later.” GPU utilization is near zero and the bill is not. Recommendation jobs already show a CPU family is enough. What should they do?
• A ferry-gate scoring endpoint is over its p99 budget. ModelLatency is about 40 ms, and OverheadLatency dominates. Staff want a larger instance. What should they do first?
• A trail-permit endpoint is slammed with invokes, but DesiredInstanceCount never moves. Target tracking was already configured. What should operations check?
• Auto scaling wants six ml.c5.xlarge hosts for a snow-desk scorer. The account quota is four. CloudWatch shows failed scale activities, not a bad model. What is the problem?
• A meter-space wait scorer already runs on AWS Lambda. After it is warm it meets latency, but the first call after idle nights misses the SLA. What ops control should they use?
• Four park districts share one SageMaker account. Cost Explorer shows a single unallocated endpoint bill. What must they apply before any cost tool can split that spend?
• The monthly SageMaker bill doubled. Tags are already on the resources, but nobody knows whether a forgotten training job, a 24×7 GPU endpoint, or data-capture storage drove the increase. What should they use?
• A compost-yard team will not watch Cost Explorer every day. They need a threshold alert before the month closes. What should they create?
• Several municipal endpoints sit idle on nights and weekends. Staff want a cost and utilization finding they did not have to build by hand. What should they consult?
• A 24×7 scoring fleet and a kill-and-retry nightly feature fleet are both paying On-Demand. How should they match purchasing options?
• Finance needs the official invoice-level breakdown for the ML account (credits, Savings Plan coverage, unused reservations), not only a Cost Explorer chart an engineer bookmarked. What should they use?
• Every intern role can run s3: on the bucket that holds the production model.tar.gz and captured inference payloads. What should they do?
• A harbor surveyor’s IAM user is attached as the role the training job assumes. When that user is deactivated, jobs fail; while it is active, the job inherits console-admin rights. What should they use?
• Security wants a data-scientist persona that can train and a second persona that can only deploy, without hand-writing every action from scratch. What should they use?
• A model-artifact bucket looks private in the ACL, but its bucket policy still allows Principal "" on a report prefix and allows non-TLS gets. IAM roles already look tight. What should they apply?
• Twelve operators each have a unique inline policy copy. When endpoint-update must be removed from analysts, half the copies are missed. What should they use?
• Training jobs and the production endpoint still start with default-VPC public IPs. Staff need to isolate the ML system, not attach an already-designed VPC. What should they design?
• Notebooks in a private subnet must call SageMaker APIs without a NAT path to the public SageMaker endpoint. What should they create?
• Studio and Model Monitor jobs run in a custom VPC. Schedules fail because the job cannot reach Amazon S3 or CloudWatch without a public hop. What should they add?
• A water-plant endpoint’s security group allows 0.0.0.0/0 on the inference port “because the VPC is private.” A compromised host in another subnet can invoke it. What should they restrict?
• The data lake is already encrypted. The production endpoint’s EBS volumes and the Model Monitor output prefix still use the AWS-managed default, which agency policy forbids for live scoring data. What should they apply?
• A CodeBuild stage and a BYOC container still embed a database password for a permit-desk feature lookup. Where should that credential live?
• A regulator asks who changed the production endpoint last quarter and whether encryption settings drifted. CloudWatch metric dashboards cannot answer that. Which pair should they use?
• The CodePipeline and CodeBuild role can iam:PassRole to any role and sagemaker: in every account. What should they do?
• A buildspec exports an IAM user’s access key to push to Amazon ECR and call SageMaker. The key lives in the CodeBuild project environment. What should replace it?
• A custom training image can still reach the public internet and an unexpected webhook. The job must not initiate outbound calls. What should they enable?
• An inference pipeline runs two containers on the same host. Agency policy requires encryption in transit between those containers, not only TLS from the client to the endpoint. What should they enable?
• An application role receives AccessDenied on sagemaker:InvokeEndpoint (VPC and KMS look fine). What should they debug?
• Data capture now writes raw request bodies. A developer later finds national IDs in the production capture prefix. What should they use to find that PII in live artifacts?
• An auditor asks for the immutable record of what a fire-inspection model is for, its risk rating, and its last evaluation. The Model Registry has versions but no intended-use document. What should they create?
• A records request asks which exact training set and job produced last March’s production model. Registry version notes are incomplete. What should they use?
• Experiment notebooks and the production endpoint share one account and one IAM neighborhood. Sandbox principals can still reach live artifacts. What should they use to isolate production ML?
• Last year’s review required KMS and VPC isolation. A new endpoint launched this month with neither, and nobody noticed until the audit. What should detect that drop continuously?