ML Model and Foundation Model (FM) Development

MLA-C02 · 48 questions

• A city's 311 department is evaluating several Amazon Bedrock foundation models for a resident-service chatbot that must handle long conversation histories and respond fluently in the languages spoken by the city's resident population. What should guide the department's model selection?
• A county permitting office wants an assistant that answers fee-schedule questions, but permit fees change every quarter. The team is deciding whether to fine-tune a foundation model on the current fee schedule or use retrieval-augmented generation against a fee-schedule document. Which approach better fits the requirement that answers stay current as fees change?
• A transit authority needs to forecast daily bus ridership from years of structured, numeric ridership history. A data scientist is deciding between a traditional gradient-boosted model and a foundation-model-based approach. Which consideration should drive the decision?
• A school district is deciding whether to use Amazon Comprehend's built-in text classification or train a custom text classifier to route parent complaint emails to the correct department. The categories map closely to generic complaint types Comprehend already models, and the district wants a solution quickly. What should guide the decision?
• A water utility engineering team is choosing between a SageMaker built-in anomaly-detection algorithm and a fully custom training script for identifying anomalies in pipe-pressure sensor readings. The readings follow a standard time-series pattern that closely matches the built-in algorithm's intended use case. What should guide the decision?
• A public housing authority designing a tenant-handbook Q&A assistant must pick between a single consolidated knowledge index and multiple topic-segmented indexes for its RAG architecture, given a handbook that spans maintenance requests, lease terms, and grievance procedures. Which pattern better fits documents that span several distinct topics with different retrieval precision needs?
• A county clerk's office digitizing decades of scanned filings must decide whether Amazon Textract or a custom-built OCR pipeline better fits its structured-form extraction needs. The forms are common layouts that Textract's form and table extraction is designed to handle. What should guide the decision?
• An emergency management agency assessing storm-damage photos submitted by residents through a mobile app must choose between Amazon Rekognition and a custom-trained computer vision model. The damage categories are highly specific to the agency's local infrastructure and don't map to Rekognition's general-purpose labels. What should guide the decision?
• A city parking enforcement team is selecting a modeling approach for classifying violation photos, and officers need a classification result in well under a second at the point of issuing a citation in the field. What should the team weigh most heavily when choosing between approaches?
• A 311 assistant team finds their foundation model doesn't reliably recognize municipal-specific vocabulary and department names used across the city's service catalog, even though it already follows instructions well. The team is deciding between instruction fine-tuning and continued pre-training as the customization strategy. Which should they choose?
• A county courts department needs to transcribe recorded hearings for the public record and is deciding between Amazon Transcribe and a custom-built automatic speech recognition model. The audio is standard courtroom speech in a supported language, and the department wants to avoid the cost of training and maintaining a speech model. What should guide the decision?
• A public health department is building a symptom-report search tool and is deciding whether a pretrained embedding model is sufficient or whether training custom embeddings on its own clinical notes is justified. Early testing shows the pretrained model already captures semantic similarity between symptom descriptions well. What should guide the decision?
• A city IT team is weighing a single large foundation model against a cascade of smaller, specialized models for a resident-facing assistant, trying to balance response quality against inference cost. What is the key tradeoff the team should reason through?
• A permitting office is deciding between fine-tuning and RAG for a building-code compliance assistant. The relevant regulations rarely change, and the office's top priority is maximizing answer accuracy against the source material rather than minimizing how often the system needs updating. Which approach better fits this priority?
• A county benefits agency must pick a SageMaker modeling option for detecting fraudulent application submissions, weighing training time against detection accuracy under a tight rollout deadline. What should guide the choice?
• A public library system wants to gauge patron feedback sentiment from survey comments and is deciding between Amazon Comprehend's built-in sentiment analysis and training a custom NLP sentiment model. The sentiment categories are the standard positive, negative, neutral, and mixed classes. What should guide the decision?
• A water utility trains an XGBoost model to predict water-main failures and wants to search hyperparameters efficiently rather than manually trying combinations one at a time. What should the team use?
• A county housing agency needs to score housing-assistance eligibility using its own custom feature engineering logic that doesn't match any SageMaker built-in algorithm's expected input format. The team writes a custom SageMaker script-mode training job. What does this decision reflect?
• A transit authority customizes a Bedrock foundation model using prompt-based customization so it consistently answers schedule questions in the agency's preferred tone and format, without retraining model weights. What does this choice reflect?
• A school district is training an attendance-prediction model and wants to cut training time without hurting accuracy. The team configures the training job to halt automatically once validation loss stops improving for several consecutive epochs. Which outcome does this early-stopping configuration most directly produce?
• A public health department is training a large chest X-ray screening model, and the image dataset keeps growing each month as new scans are collected. Single-instance training runs are taking longer every cycle and are starting to delay releases. Which change best keeps training time manageable as the dataset scales?
• A 311 assistant team is training a small service-request classifier on a limited labeled dataset. Training accuracy keeps climbing toward near-perfect while validation accuracy has plateaued far lower and is starting to drop. Which change best addresses this pattern?
• An emergency management agency combines predictions from several independently trained flood-risk models by averaging their outputs, aiming to make forecasts more reliable heading into storm season. Which statement best explains why this ensembling approach tends to improve reliability?
• A city permitting office is tuning a document classifier's training job across epoch count and batch size to reach acceptable accuracy while keeping training cost under control. The team notices that a larger batch size lets each epoch finish faster but the model needs more epochs to reach the same accuracy. Which conclusion best fits this tradeoff?
• A county courts department wants to improve retrieval quality for a case-law search assistant. Generic embeddings return semantically related but legally imprecise matches, often conflating distinct legal terms with similar everyday meanings. The team decides to fine-tune the embedding model on legal terminology. Why is this the appropriate lever for this problem?
• A public housing authority wants a foundation model to broadly understand its internal housing-policy terminology, procedures, and case history before it is used for a casework assistant. The team runs continuous pre-training on its internal policy corpus rather than fine-tuning on a small labeled task dataset. What distinguishes this choice from fine-tuning?
• A city is deploying a lobby kiosk chatbot to answer simple resident questions and wants a cheaper, faster model than the large foundation model currently used for more complex casework elsewhere. The team trains a smaller student model to reproduce the larger model's behavior on this narrow set of simple questions. What technique does this describe?
• A water utility fine-tunes a foundation model on new sensor terminology and maintenance procedures. After fine-tuning, staff notice the model has become excellent at sensor questions but has lost much of its earlier ability to hold a general conversation about unrelated topics. Which approach best mitigates this pattern?
• A county IT team wants to flag irregular patterns in utility billing records without having labeled examples of what billing fraud or errors actually look like. They plan to use a SageMaker built-in algorithm suited to this kind of unsupervised outlier detection. Which consideration should most directly guide their algorithm choice?
• A 311 chatbot team wants to improve handling of multilingual resident queries and is weighing adding few-shot examples through prompt engineering against fine-tuning the model. Early testing shows the base model already understands the languages well but occasionally misapplies the city's specific response format. Which approach fits this situation best?
• A transit authority's RAG-based schedule assistant sometimes returns answers built from irrelevant retrieved passages, and sometimes misses relevant schedule details that were split across two separate retrieved chunks. The team tunes chunk size and top-k retrieval count to address this. Why are these the right parameters to adjust here?
• A permitting office is training a zoning-classification model and chooses to write a custom PyTorch training script run through SageMaker script mode instead of using a SageMaker built-in algorithm. The model needs a nonstandard architecture with a custom loss function tailored to how zoning violations are weighted. Which factor best justifies this choice?
• A city water-quality modeling team has had several staff transitions over the past year, and each new hire has struggled to reproduce prior experiment results because parameters and outcomes were tracked informally in spreadsheets. The team adopts MLflow on SageMaker AI to log experiment parameters and results going forward. What problem does this most directly solve?
• A county benefits office wants to choose between several candidate foundation models for summarizing eligibility determinations before wider rollout. The team uses Bedrock Model Evaluation to run each candidate against a defined summarization task and compare results. What does this evaluation step most directly provide to the team?
• A 311 chatbot team measures response quality by comparing generated answers against reference answers using BLEU and ROUGE scores. A reviewer notices these scores penalize a response that used different wording but conveyed the exact same information as the reference answer. What does this reveal about BLEU and ROUGE as evaluation metrics?
• A public health department evaluates generated symptom-report summaries using BERTScore-based semantic similarity instead of exact word-overlap metrics. A reviewer asks why the team moved away from lexical-overlap scoring for this task. What is the most accurate justification?
• A city establishes a performance baseline for its permit-approval-time prediction model by recording its accuracy on a representative validation set at launch. Several months later, monitoring shows live predictions have drifted noticeably from that recorded baseline. What does this drift most directly indicate the team should do next?
• A transit authority runs a new ridership-prediction model as a shadow variant, receiving live traffic alongside the current production model but with its predictions withheld from riders, before deciding whether to promote it. Why is this evaluation approach useful before promotion?
• A housing authority must justify to a review board why a specific application for rental assistance was denied by its scoring model. The team applies model explainability techniques to produce feature-level reasoning for that individual prediction. What does this evaluation approach provide that overall accuracy metrics do not?
• An emergency management team debugging a flood-prediction model notices training loss oscillating wildly and failing to steadily decrease across epochs, rather than converging smoothly. Which cause is most plausible and directly addressable during training configuration?
• A school district adds a human-in-the-loop review step where staff must approve FM-generated individualized education plan summaries before they reach families. Given how high-stakes and legally sensitive these summaries are, what is the most important design consideration for this review step?
• A county courts system uses an LLM-as-a-judge approach, where a separate foundation model scores whether chatbot-generated legal-information responses are accurate and complete, to evaluate output at a scale human reviewers could not sustain. What is the most important limitation the team should account for when relying on this approach?
• A city evaluates its benefits-scoring foundation model for differences in outcomes across demographic groups before wider rollout, beyond just checking its overall accuracy score. Why is this bias-detection step necessary in addition to overall accuracy?
• A water utility uses Bedrock Prompt Management to version and A/B test prompt variants for its automated leak-report classifier, rather than editing prompts directly in application code each time a change is tested. What advantage does this approach provide?
• A 311 resident-services team's retrieval-augmented assistant answers questions from a knowledge base that grows and changes every week. Six months after launch, more answers cite outdated or missing policy text even though the generation model itself hasn't changed. What should the team do to catch this kind of drift going forward?
• A permitting office runs a hybrid pipeline: a traditional classifier first sorts an application into a permit type, then a foundation model drafts a plain-language explanation for the applicant. How should the team evaluate this pipeline's performance?
• Before committing to a foundation model for production case-management summaries, a county IT team wants to compare several candidate Bedrock FMs on that specific summarization task. How should they use Bedrock evaluations here?
• A transit authority's RAG-based schedule assistant scores well on BLEU and ROUGE against reference answers, yet riders keep reporting confidently wrong schedule times in its replies. What should the team add to its evaluation process to catch this?