CT-AI practice questions
ISTQB · CT-AI · 300 questions
Validates tester-level ability to design and execute tests for AI-based systems, including machine learning systems and generative AI, covering AI quality characteristics, input data testing, model testing, and ML development testing using techniques such as metamorphic testing and red teaming.
This course contains the use of artificial intelligence.
About the CT-AI exam
- Time allowed
- 1 hour
- Questions
- 40
- Passing score
- 29 of 44 points
- Format
- Delivered by accredited ISTQB exam providers; 25% extra time (75 minutes) for non-native language speakers
Exam details published by the vendor, checked 25 August 2026. Vendors change fees and formats without notice — confirm on the vendor's own page before you book.
Practice Quizzes
Test your knowledge with standard 20-question practice sets.
Quiz 1
Quiz 2
Quiz 3
Quiz 4
Quiz 5
Quiz 6
Quiz 7
Quiz 8
Quiz 9
Quiz 10
Quiz 11
Quiz 12
Quiz 13
Quiz 14
Quiz 15
Browse by Domain
Study specific topics at your own pace.
Introduction to Artificial Intelligence · 45 questions
- A harbor office opens lock gates from a coded if-then timetable. A neighboring marina scores “likely delay” from years of tide and traffic logs and never wrote those rules. Which statement is correct?
- A compost-site scale house wants reject/accept to be identical every time the same truck weight and moisture reading arrive. Their vision sorter, given the same hopper photo twice, can emit slightly different grade labels. How should a tester classify those behaviors?
- A credit-union lending desk must tell a member why a small-business loan was declined. One option is a short decision tree on five application fields; the other is a many-layer network with millions of internal weights. Which option creates the black-box explainability problem for testers and auditors?
- A municipal snow-route office used to ship one fixed plow-priority table each November. The replacement system keeps changing those priorities as new storm telemetry arrives. What distinction should the tester recognize?
- A parks department hears three pitches: a coded irrigation timer, “any system that acts intelligently,” and a moisture predictor that improved only because it was fitted on historical soil logs. How should those three be nested?
- A ferry-queue estimator must cope with fog, late trains, and festival crowds that no one listed as coded exceptions. Stakeholders ask why the team did not just write more branches. Why is a probabilistic, pattern-based AI-based system chosen here?
- A kelp farm’s camera only estimates frond density and is marketed as “general intelligence for the whole marina.” How should a tester classify the deployed system?
- A fish hatchery’s count-from-video model is asked to start writing the weekly feed-purchase emails. Operations assumes last year’s vision work will “just handle words too.” What should the tester know?
- A foundation-model clerk’s assistant drafts ordinances and answers permit questions and is labeled “frontier” on the vendor slide. A councilor treats that label as proof of general AI. How should the tester place frontier AI?
- A city CIO claims the new assistant can perform most intellectual work any staffer can, across unfamiliar departments, without being prepared again for each desk. What claim is that, and does such a system exist today?
- A risk briefing describes an AI that keeps improving itself without human control, exceeds both human intellect and general AI, and — if the jump ever happens — is named as the technological singularity. Which term matches that description?
- A workshop argues that super AI is impossible unless the system is on the public internet. Another participant says a closed facility could still host it, though a network would widen its reach. Which statement is correct?
- A grain elevator wants a moisture-grade predictor that is not handed a written formula; it should form the mapping from labeled historical samples. Which technology is that?
- A vineyard night-shift board shows three pilots: nights labeled frost / no-frost for a fan controller, unlabeled cellar aroma readings grouped into clusters, and a lock-gate agent that earns a reward for fewer vessel waits. How should those three be classified?
- An orchard desk has still photos of leaf spots, a river desk has hourly gauge readings, and a permits desk has long statutes whose early clause changes a later one. Which deep-learning families match those data shapes?
- A community-radio archive wants volunteer show notes grouped by topic and named guests. A glacier desk wants crevasse photos flagged when the split widens. Which technologies match those two requests?
- A canal office uses a “somewhat high” water-level controller, a barge router that searches a lock network for a cheaper path, and a cheese-aging cellar that fires a coded knowledge base of if-thens. Which non-ML AI technologies are those?
- A warehouse robot chooses a pick path, replans when an aisle is blocked, and acts without waiting for a human step-list. A second system only labels a bin photo as “full” or “empty.” Which system is agentic AI?
- A QA kickoff slide lists linear regression, decision trees, support-vector machines, random forests, Bayesian models, and neural networks as candidate techniques for a hatchery grader. How should a tester treat those names?
- A tourism board wants paragraphs for a visitor guide and painted poster scenes from a short brief. A second vendor only sorts last year’s photos into “harbor / mountain / festival.” Which request is generative AI?
- A ceramics studio’s research bench has one network inventing extra kiln-defect pictures and a second network trying to spot which pictures are fabricated. Which generative approach is that?
- A print shop’s poster tool begins with a field of noise and step-by-step removes that noise until a festival illustration appears. Which generative approach is that?
- A municipal clerk’s drafting tool still respects a definition that appeared two pages earlier when it writes the closing clause. What mechanism should the tester tie to that long-range coherence?
- A harbor-authority communications desk worries about a fabricated video of a “channel closed” announcement spreading before any official notice. What GenAI societal risk should a tester be aware of?
- A county legal office sees first-draft ordinance language and routine medical-chart summaries being produced by a generative tool. Which syllabus societal concern should the tester recognize?
- A coastal-notice desk wants one system that reads tide charts and writes the public bulletin. Training that large multimodal stack burns a lot of electricity, and the vendor started from a broad pretrained base then specialized it. What should the tester connect?
- A trail-head kiosk transcribes hikers’ spoken reports on a low-power board, but the model was fitted for two weeks on a data-center cluster. How should a tester compare the hardware?
- A hatchery lab argues their office PCs “clock higher, so they must beat the graphics boards at fitting the grader.” What hardware comparison should the tester apply?
- A lighthouse edge box must cut the number of bits used for each arithmetic value so the chip stays cooler, cheaper, and less hungry for bandwidth. Which hardware trait is that?
- A river buoy carries a purpose-built AI chip (an ASIC or system-on-chip) that does in-memory, multi-core inference, while the model itself was fitted in a cloud hall. How should the tester place that hardware?
- A research pier is evaluating processors that abandon the usual stored-program layout and instead mimic neuron-like structures. Which hardware should a tester recognize?
- A small maritime museum can subscribe to a pretrained captioning service and go live next month, or fund a private model that obeys its accession rules but needs scarce specialists. What trade-off should the tester compare?
- A clinic wants to sketch a compact decision tree on a staff laptop so patient fields never leave the building, then later fit a larger network on a public cloud that already has GPUs and a usage bill. How should a tester compare those development options?
- A regional hospital keeps note-redaction and private-data prep on a private cloud, bursts the heavy fitting job onto a public cloud, and calls the mix hybrid. Which traits should the tester match?
- An allergy-alert model could run on clinic phones (privacy, no hosting fee, weak hardware), on a dedicated on-site server (up-front cost, more control), or on elastic public hosting. What is the tester comparing?
- A canal authority fitted a mid-size model on a rented cluster but now wants to host the running model on its own lock-house servers because of a new privacy rule. What should the tester conclude?
- A cooperative’s toolkit loads and cleans harvest logs, lets staff pick a tree versus a net, iteratively fits internal parameters, scores precision on unseen weeks, and exports a file a kiosk can run. Which five framework function groups are those?
- A junior analyst wants a high-level interface so a frost-fan prototype exists by Friday. A specialist wants a low-level interface so every layer and operation can be specified. What trade-off should the tester compare?
- A transit desk can adopt a general-purpose framework that covers several problem types, or a specialized framework aimed at spoken announcements. What should drive that selection?
- A lighthouse lab lists what it cares about: a gentle interface for a short prototype, room to configure a harder model later, the staff’s current skill, whether the result must run on a constrained buoy, and how active the user community is. How should a tester treat that list?
- A city council wants a human-centric, internationally cooperative baseline before it drafts a local AI ordinance. Which instruments should a tester recognize?
- A clinic’s triage assistant that can affect safety and fundamental rights is placed in a high-risk band of a risk-based statute. A festival chatbot that only writes slogans sits much lower. What should the tester explain?
- Counsel warns that a breach of the EU-style act can be charged as a share of worldwide turnover, while some governments outside that act prefer lighter rules to keep experiments moving. What should the tester contrast?
- A QA lead must show a regulator that the team used recognized guidance for testing AI-based systems. Which syllabus-named standards should the tester point to?
- A port authority asks why any of these rules exist, which quality-model number testers will meet next, and whether last year’s checklist is good forever. What should the tester answer?
Quality Characteristics for AI-Based Systems · 23 questions
- A water-treatment lab scores its algae-bloom predictor two ways: one checklist about the built product and another about use in the plant. How should a tester place ISO/IEC 25059?
- A tile-glaze line used to treat every wrong shade call as a failed release. The new vision grader is probabilistic, so the quality plan now names an acceptable band of wrong calls instead of demanding zero misses. Which 25059 characteristic is that?
- A greenhouse climate planner is already in production. When a storm front changes humidity, it retargets vent and mist setpoints on its own, without a code drop. Which 25059 characteristic is that?
- A ski-patrol dispatcher can hit a “hold all” key and take route assignment away from the advisor while the next sled is still being staged. A consultant calls that “usability.” Which 25059 characteristic is that?
- A university archives desk must tell researchers which captioning model is live, where its training corpus came from, and where the model card lives. The same information is also meant to support a user’s sense that the service is understandable. Which 25059 characteristic is that?
- A recycling-belt sorter still meets its agreed sort-error band when a share of frames are overexposed, a few labels in the feed were skewed, a jam shakes the camera, or an operator feeds a bag the wrong way. Which 25059 characteristic is that?
- A hydroelectric gate advisor is about to recommend opening a spillway. The night operator must be able to stop that act in time so the river below is not put at risk. Security, not the control-room “ease of use” checklist, owns that freeze. Which 25059 characteristic is that?
- A municipal hiring screen must not treat named groups differently on the agreed fairness metric, must drop fields that identify applicants, and must leave the final offer with a human panel. Which 25059 characteristic is that bundle?
- Two freeze buttons sit on the same quarry-loader console. One lets a supervisor take steering back so the shift can finish the load. The other exists only so an operator can abort a move that would endanger a walker in the pit. How should those two be classified?
- A rare-book lab asks why the test plan now talks about error bands, post-go-live retargeting, seize times, model-card visibility, dirty-input endurance, hazard freezes, and a fairness gap. What should the tester see?
- A mine-conveyor stop is supposed to “keep hands off the belt.” The conventional interlock had numbered requirements down to code. The vision add-on is specified mostly as “learn from last year’s incident clips.” What safety challenge should the tester explain?
- A bakery oven-guard model, given the same thermal frame twice, can emit two slightly different “pull tray / leave tray” commands because of small input jitter or internal randomness. A safety assessor wants a guarantee of one exact command. What obstacle should the tester explain?
- A wildlife-crossing barrier was safety-tested in March. By August it has been updating itself from new trail-camera nights, and the March test no longer describes what it does. What should the tester explain?
- A ski-lift hold model dropped a chair line and nobody can reconstruct the reason from the weights. An explainability add-on might sketch a local reason, but it is not widely available and the vendor warns it can slow the hold decision. What safety problem should the tester explain?
- A medical-device brake that uses a vision cue is treated as a safety component under a risk-based statute and therefore draws extra development and test duties. The plant’s older functional-safety standard never names AI; a sibling standard still forbids it in the safety function. What regulatory picture should the tester explain?
- A paper-mill trip system has a conventional pressure interlock beside a new acoustic “web-break” model. The assessor lists five extra AI headaches: fuzzy goals in the sound archive, non-repeatable calls, a model that keeps learning, a decision no one can narrate, and rules that changed twice this year. What should the tester recognize?
- A cider-mill bruise grader cannot be accepted with “never wrong.” Legal and ops instead want a numeric error band, a timed seize, and a fairness gap, each with a threshold. What typical shape of AI acceptance is that?
- A herbarium’s leaf-ID helper may mis-label at most 4% of a held-out tray of rare sheets, and recall on those rare sheets must stay at or above an agreed floor. What kind of acceptance criteria are those numbers?
- A greenhouse vent planner must retarget humidity setpoints within eight minutes after a 20-point outside-humidity jump, without a technician pushing a new build. What acceptance criterion is that?
- A radio-archive loudness advisor must yield to a board-op “take the fader” key within two seconds, and must fully mute itself if the key is held and no human ack arrives. What acceptance criterion is that?
- Every scored insurance-claim file must carry the live model’s unique version id, a link to its model card, and the vintage of the training corpus, matching the firm’s disclosure rule. What acceptance criterion is that?
- A sawmill knot classifier must keep its agreed error band when 12% of frames are motion-blurred, when a share of labels in a dirty feed are skewed, and when the mill lights flicker for half a minute — it may degrade to a slower, coarser mode, but it must not crash or silently abandon the band. What acceptance criterion is that?
- Three acceptance lines sit on one dam-and-hiring program: any “open spillway” advice stays frozen 45 seconds so a licensed operator can cancel it; a hiring screen’s agreed fairness metric may not exceed a stated gap across named groups; the conventional over-pressure interlock must still trip even if the acoustic model is unsure. How should those three be mapped?
Machine Learning · 53 questions
- A cheese-cave shop has three pitches: shelves already tagged “ready / wait,” a heap of unlabeled aging logs to be grouped by similarity, and a robot cart that is praised or fined after each aisle choice. How should those three be assigned?
- The same cave wants one model that stamps each wheel “sell / hold / discard” and another that estimates days-to-peak in a decimal. Both use labeled history. How should that supervised work be split?
- A test lead hears “we need regression on the mash-temperature predictor” and books a full re-run of last sprint’s functional suite. The data scientist meant a continuous-value supervised model. What distinction should the tester keep?
- A mountain-hut shop has unlabeled till rolls. One request is “put guests into groups that look alike.” The other is “which snack tends to appear with which fuel canister.” How should those unsupervised requests be assigned?
- A rope-tow start gate has no labeled “correct start” file. A cart learns by inching forward, receiving a plus when skiers load smoothly and a minus when the line bunches. Which ML form is that?
- The rope-tow cart is rewarded for “gate cycles per hour.” It learns to slam the gate whether or not a skier is ready. What RL challenge should a tester understand?
- A seed-bank desk has labeled germination outcomes, a drawer of unlabeled packet photos, and a robot arm that can only be scored after it places a packet. How should the tester pick the ML form?
- A paper mill wants a tear-class stamp (labeled), a grouping of unlabeled vibration weeks, and a reel-cart that is scored after each dock. What should the tester refuse?
- A beekeeper co-op argues for three weeks about whether the model exists to flag mite risk or to forecast honey weight, and only then writes the ML functional-performance numbers the model must meet. Which workflow activity is that?
- The co-op picks a development toolkit from the job and the acceptance bar, then picks an algorithm (from a library, not hand-coded) that matches the labeled mite sheets they actually have. Which two workflow steps are those?
- Before any fit, the mite sheets are acquired, cleaned, and featured, and the prep steps themselves are checked. A volunteer wants to “just train and see.” Which workflow activity is required?
- One setting is “how many layers / how deep is the tree” (structure passed into the algorithm). Another is “how many passes over the mite sheets” (how training itself is run). How should a tester name those knobs?
- The co-op fits a tree, a margin classifier, and a small net on the same training slice, then scores them on a validation slice against the agreed metrics and keeps the best pair of algorithm plus training data. Which workflow step is that comparison?
- After the validation scores arrive, the team nudges layer count and pass count, retrains, and scores again. They keep calling that loop “the test.” Where should a tester place that loop?
- Once a model looks good enough, a slice that never guided a nudge is used to check the agreed ML functional-performance criteria and is compared with the validation numbers. If the held-out score is much worse, the team returns to model generation or even to data prep. Which workflow step is that?
- The chosen mite model is re-engineered with its data pipe for a hut kiosk and a small cloud API. Acceptance is checked again on that packed form. Which workflow step is that?
- Once live, the model either scores a nightly batch of yard sheets or answers a ranger on demand. It sits inside a larger yard system, so conventional integration, system, and acceptance still happen. Over the season its scores may walk away from the agreed bar, so the team re-evaluates and may rebuild. Which closing workflow should the tester summarize?
- A herbarium cannot afford to train a leaf-net from empty weights. It starts from a network already fitted on a broad plant-image corpus, then spends a much smaller labeled set on its own sheets. What alternative should the tester summarize?
- The herbarium can extra-train the whole net, only the layers near the output, or a few added layers, then re-check ML functional performance and extra-train again until acceptance is met. What should a tester understand about those choices?
- Adapting a “maple-leaf vs oak-leaf” net to “beech vs birch” is likely to work. Asking that same net to score recorded bird-call accents is not. What should the tester summarize?
- A clock-tower help desk keeps a binder of winding procedures. Those pages are turned into a searchable store, the closest pages are stuffed into an enlarged prompt, and the language model’s weights are never changed. Which approach is that?
- The tower desk both extra-trains a small language model on last decade’s logbooks and still retrieves tonight’s weather bulletin into the prompt. What should the tester know?
- The bought leaf-net already favored glossy studio photos and failed on frost-bitten sheets. After fine-tuning on the herbarium’s own trays, those habits are still there. What risk should the tester summarize?
- A soap factory’s scent-batch project must first decide it needs numeric vat logs plus a few still photos, pull them from the historian and a bench camera, and have two chemists tag “in spec / off.” Which data-preparation activity is that?
- The vat log has smashed rows, the same batch twice, a one-off 900 °C spike, empty pH cells, and a column of worker names. Which preprocessing activities map to those defects?
- One vat reports temperature in a 0–1 scaled column; another still stores raw °C. A third dumps a free-text note where the model needs a fixed-width row. Which preprocessing work is that?
- The factory has too few off-spec batches, so it adds rotated stills, a few synthetic off-spec rows, and some deliberately noisy frames so the later model is less brittle. Which data-prep activities are those?
- The historian holds ten years of vats. Compute and calendar only allow a representative subset this sprint. Which preprocessing activity is that?
- Twenty sensor columns exist; three of them actually move the “in spec” call, and two others are near-duplicates of those three. The team keeps a smaller, informative set. Which activity is that?
- Beside the cleaning work, a chemist sketches weekly pH clouds and spots a week when the probe stuck high. Which parallel activity is that?
- The soap team reorders cleaning and feature steps twice, skips anonymization on a lab-only trial, then insists the live vats must show the same distributions and ranges as the training vats — even if the live pipe is simpler for speed. What should the tester require?
- A yarn-dye house randomly draws three equivalent slices from one representative lot: one to fit the shade model, one to score and nudge it, one held back as the final judge. How should those slices be contrasted?
- The dye-house slices were all drawn from winter lots. Spring lots are more pastel. What must be true of the training, validation, and test slices?
- With plenty of lots, the dye house still must decide how much data each slice gets. Which factors should drive those volumes rather than a single recited percentage?
- Only a short season of lots exists. Cutting it three ways leaves too little to fit, and the shade model underfits. What risk and response should the tester recognize?
- The remaining pool is cut into k parts (often 5 or 10). Each part is the validation slice once while the others train; scores are averaged; the winning hyperparameters then refit on the whole pool and are judged once on the holdout. Folds are usually random and often stratified when classes are thin. How does that differ from a single static validation slice?
- A second dye house is so short on lots that it skips the holdout and quotes the average fold score as the official number. What should the tester know?
- A kiln-crack report prints a 2×2 of predicted versus actual “crack / sound.” One vendor swaps which axis is predicted. What must the tester still recover from either layout?
- A kiln-crack matrix counts 42 true cracks called crack, 48 true sound tiles called sound, 7 sound tiles called crack, and 3 true cracks called sound. What is accuracy as (TP + TN) / all × 100%?
- A candle-wick inspector’s false “defect” calls stop the line. The matrix is 21 true defects called defect, 7 sound wicks called defect, 2 missed defects, and 70 sound wicks called sound. What is precision, and what does it mean?
- A cave-bat-count model that misses an actual roost night is the expensive error. The matrix is 15 true roosts called roost, 5 missed roosts, 4 false roost calls, and 76 true quiet nights called quiet. What is recall (sensitivity), and what does it mean?
- A tea-blend grader posts precision of 90% and recall of 60%. What is F1 as 2 × (precision × recall) / (precision + recall), and why is it not the arithmetic mean?
- A glass-studio “bubble / no-bubble” set is 190 no-bubble tiles and 10 bubble tiles. A model calls every tile no-bubble: 0 true bubbles found, 10 bubbles missed, 0 false bubble calls, 190 true no-bubble calls. Why does 95% accuracy mislead, and which named metric should the tester also compute?
- A print-shop registration model never hits 100% on accuracy, precision, recall, or F1. Stakeholders treat that as a failed build. What should the tester explain?
- Two bicycle-share “needs-service” models each score 100 bikes. Model A: 20 true positives, 15 false alarms, 5 missed bad bikes, 60 true negatives. Model B: 20 true positives, 5 false alarms, 15 missed bad bikes, 60 true negatives. Missed bad bikes staying in service is the costly error. Which model matches that cost story, and why?
- A soapbox-derby “legal / illegal ballast” demo uses a single-layer perceptron that can separate the two classes only if a straight line in feature space can do it. How should a tester place that structure?
- A weather-balloon photo hits an input layer of pixel-like values, passes through one or more hidden layers of nodes, and leaves an output layer as an “icing-risk” score. In a fully connected net, each node talks to every node in the next layer, and layer widths may differ. What structure should the tester name?
- Each hidden or output node forms a weighted sum of the previous layer’s activations, adds its own bias term, and pushes the total through a non-linear activation function. Different functions give different activations. What per-neuron computation should the tester recount?
- Weights start small and random (biases often at zero). A balloon photo goes forward, the output is compared with the known icing label, the error (loss) is sent back to nudge weights and biases, and one full pass through the training set is an epoch. Training stops when the output is good enough. What working cycle is that?
- A fairness reviewer hears “bias” on a slide about hidden nodes and opens the Chapter 5 bias-test checklist. What distinction should the tester keep?
- A tile-factory net has many nodes that never produced an activation above the tester’s chosen cut during the test set. Which coverage measure is that, and what does a silent region suggest?
- Each node’s possible output range is cut into k sections; testers ask what fraction of those sections fired on the test set. Which coverage measure is that?
- During test, some nodes fire higher than any training maximum or lower than any training minimum. Which coverage measure is that, and what caution should the tester remember?
Testing AI-Based Systems · 52 questions
- A district-heating leak detector ships with a deep network whose weights and biases are frozen at go-live and can change only if the team retrains. A locker-room climate agent keeps changing setpoints from a live reward after opening night. Which system is locked, which is adaptive, and which is easier to test?
- A sidewalk-heave classifier is locked in the field, then the vendor ships a new fitted snapshot every quarter. Testers still have last quarter’s expected-result file. How should they treat the update?
- A public-pool turbidity classifier is locked, yet the same frame scored on two parallel graphics nodes can differ by a few thousandths. Stakeholders call that a defect in the lock. What should the tester recognize?
- Testers run a long weekend of simulated demand through a bike-share rebalance agent that learns from a reward. Monday’s routing no longer matches Friday’s. What testability problem is that?
- After a storm week a food-bank restock agent invents a transfer habit testers never enumerated. The board asks for the pre-written cases that cover it. What should the tester say?
- A streetlight-outage agent adapts after a heat wave. Testers keep an automated suite aimed at core routing and safety stops and rerun it whenever the agent changes a lot, plus a watch that the miss rate has not crossed a stated band. What pattern should the tester pick?
- A town-clerk minutes summarizer does not change weights during a sitting, but the vendor drops a new snapshot every month. Where should the tester place that GenAI pattern?
- Before opening, testers put an ice-rink thickness advisor in a closed hall, shift the fake weather, and watch whether the learning rule adapts in a safe direction. What should the tester know about that rehearsal?
- An orchard frost-alert once calls a freeze a thaw. Growers want the release failed on that single night. What should the tester explain?
- A community-clinic no-show predictor is given the identical booking twice and returns two nearby scores. A conventional scheduler would have repeated exactly. Why do testers look at distributions and confidence instead of a single expected bit?
- A recycling-sort model was fitted on weekday household bags. Weekend festival bags have a different mix. What sample should testers use for a statistically meaningful verdict?
- An elevator-fault predictor is highly confident on a rare hydraulic sound and still wrong. The board wants a number they can defend. What should the tester point to?
- A watershed flood-stage classifier must show a safety threshold with high confidence to a regional board. Three vivid examples will not do. What kind of evidence should the tester use?
- A community-solar output model uses stochastic elements in the net and also emits probabilistic scores learned from years of sky logs. Either source can make two runs differ. Why is a statistical approach needed?
- A school-bus late-arrival estimator is scored like a conventional timer: each run is pass or fail. Why is that the wrong verdict for this system?
- A mountain-rescue radio-triage model emits an urgency score. Dispatchers cannot name one correct number for a garbled call. Testers agree a band and a tolerance. Which oracle workaround is that?
- A botanical-garden pest identifier started as an exploratory pilot. Requirements are still thin and keep moving. Why is the oracle hard?
- An archives handwriting transcriber is run on eighteenth-century court rolls. Checking every token by hand is not practical. What oracle challenge is that?
- A museum caption writer is judged by visitor taste. Two curators call the same caption a pass and a fail. What oracle problem is that?
- A self-learning public-pool chemical advisor keeps updating after new readings. Last month’s expected-result cards no longer match what “correct” means. What oracle problem is that?
- Port container-damage vision scores wander when the bay lighting, chill-store temperature, or camera-link lag change. Testers write those environment values into the test setup so repeats are comparable. Which oracle aid is that?
- A cooperative has two grain-moisture estimators and no lab gold label for every wagon. Which oracle workaround should the tester choose?
- Wildlife night-time audio has almost no labels. Testers train a proxy on a small labeled subset and use it to score the unlabeled pile. What oracle is that, and what risk remains?
- Testers of a public-radio playlist writer send station briefs and then judge clarity, freshness, and house-rule fit of the produced hour. They never open the weights. Which GenAI test approach is that?
- A county permit-triage assistant can be reached from a clerk console or an API, with an optional house brief and a user packet that can hold hundreds of pages. Which GenAI testing problem is that?
- A community-college placement-essay commenter is given the same essay twice: once with a high temperature and a generous token cap, once with a low temperature and a tiny cap. The two comments are not the same product. What must testers control?
- A regional-rail delay explainer keeps prior turns. A tester’s fifth question is answered as if the first rumor were still true. Which GenAI factor changed the output?
- A building-permit drawing-completeness assistant is reviewed by humans against criteria written in the requirements (missing stamp, unreadable scale, absent egress note). There is no single expected paragraph. How is the result decided?
- Generated garden wayfinding maps are scored by a vision checker. What should the tester allow, and what risk remains?
- A town-minutes summarizer is compared on a published language-understanding suite so two vendor snapshots can be lined up. Testers also log processor, graphics, memory, network, and response time on inference. What should stay in the GenAI plan?
- A clinic-advice assistant is about to leave internal QA. A lead is told to start red teaming. What is the first official step?
- Library staff want red teaming run on the lobby kiosk after lunch. Where should the tester require access?
- A permit assistant can be probed for security holes (named types only: indirect prompt injection, hidden content in retrieved documents) or for unsafe answers to an ordinary resident question. How should a tester tell those two red-team aims apart?
- A minutes summarizer has finished internal quality checks and is a week from public launch. When should red teaming run?
- Testers hold long multi-turn sessions with a rec-center coach-bot, using open exploration and a short checklist. They then group the failures and turn them into a dataset the builders can use to harden the system. Which later official red-team steps are those?
- A county assistant’s input space is huge. A small expert desk cannot cover it. Which coverage tactic should the tester choose from the official set?
- A minutes summarizer already posts a strong score on a static language suite. A sponsor wants to skip red teaming. What should the tester say?
- After a pre-launch red-team week, operations ask what still happens on the live assistant. How should the tester contrast the two teams?
- A dairy somatic-cell predictor still needs component, integration, system, and acceptance work. Which two extra ML-specific levels should the tester also name?
- Testers argue whether input data testing is only about the historic fitting set for a dairy predictor. What should the tester say it covers?
- A lead asks what ML model testing is for on the dairy predictor. What should the tester say it concerns?
- A rink kiosk has a clerk screen, a data pipe, and a radio link that are not models. Which testing still applies to those pieces?
- Testers must show that cleaned frames arrive in the shape the thickness model expects and that scores reach the kiosk screen. The same shop also buys a weather score as a service. Where do those checks sit?
- The thickness model is embedded in the full kiosk build; a smaller compressed copy is also tried. What should system testing check?
- One team checks interfaces and data exchange with an outside weather bureau in an environment that looks like the rink. Another team decides whether a bought scoring service meets the shop’s ML performance bar. How should those two be split?
- A water-utility board asks why the MLS test plan is risk-based instead of testing everything the same. What should the tester say?
- A solar-output project picked a poor algorithm, a thin evaluation approach, and a development toolkit with a known security hole. Which risk lane is that?
- A transit late-bus model was fitted on one depot’s winter logs, the live pipe drops night rows, and weekend routes were never in the pile. Which risk lane is that?
- A pest identifier misses its agreed score, looks overfitted on greenhouse shots, and flips on tiny input tweaks. Which risk lane is that?
- A second facilitator wants the MLS risk list grouped by ISO/IEC 25059 names (correctness, robustness, societal and ethical mitigation, and the rest) instead of by workflow lane. What should the tester accept?
- A lead has three named forms on the table: data-pipeline testing, adversarial testing, and a review of algorithm or model suitability. How should those attach to risk lanes without teaching later-chapter methods?
- A sponsor hears about input data testing and model testing and wants to drop component, integration, system, and acceptance work. What should the tester say?
Input Data Testing for Machine Learning Systems · 45 questions
- A rec-center scholarship screener treats one age band differently after being fitted on a historically skewed attendance pile. Which mitigation should the tester pick?
- A sidewalk-heave project bought a street-condition dump from a reseller with no chain of custody, and the files sat on an open share. Which mitigation should the tester pick?
- A water-meter anomaly pile includes a cluster of rows that look planted. Which official mitigation family should the tester name without teaching later techniques?
- A bus-stop board shows passenger counts below zero, mixed units in one column, and the word “busy” in a count field. Which mitigation should the tester pick?
- A food-bank forecast kept a locker-number column that never moves with demand. Which mitigation should the tester pick?
- A recycling-sort pile over-represents bottles, was padded with synthetic cans, has empty nights, covers only weekday routes, and never saw the high end of contamination. Which mitigation should the tester pick?
- Pest photos were tagged with vague rules and disagreeing annotators; the live pipe also drops night frames and has no check for uncontrolled edits. How should those two defects be split?
- A community-solar grant screener systematically scores one neighborhood lower. How should a tester define MLS bias here?
- A tool-library late-fee helper either inherited a decade of uneven write-offs or applies a single late-day cut that hits one age band. How should those two sources be told apart?
- Before anyone plots a chart, a tester is asked how to hunt bias introduction. Where should that hunt start?
- A rec-center dataset folder has notes on how attendance was collected, who tagged it, and which neighborhoods appear. Which bias-test approach uses that folder?
- Prep scripts quietly drop a language-preference field, and the model code treats a blank as a default majority value. Which bias-test approach finds those anti-patterns?
- Exploratory plots and clusters of a transit late-bus pile show one depot’s night shifts bunched away from the rest. What should that reveal to a tester?
- Testers send a known unbiased, representative booking set through a clinic no-show model and look for statistically significant outcome gaps across groups. Which bias test is that?
- A rec-center scholarship model is checked by identifying sensitive attributes, building plausible counterfactual rows, scoring them, and looking for a statistically meaningful flip — including combinations of attributes. What must the tester also refuse?
- A water-meter anomaly project is still in design. How should pipeline testing start?
- A pool-chemistry pipe has an ingest job, a transform script, and a sensor interface. Which testing covers those pieces?
- Testers of a grain-moisture pipe must show validation rules fire, bad rows are handled, and there is no easy hole for planted or hostile files. Which pipeline tests are those?
- Cleaned moisture rows leave stage one as “pct” and stage two still reads “percent.” What kind of defect is that?
- The assembled sidewalk-heave pipe is checked with a smoke pass, functional transform and routing cases, a load, scale, and security pass, and deliberately defective frames to see whether integrity holds. Which system-test set is that?
- Operations rebuilt the live pest-photo pipe. Testers want to know it still does the same job as the training pipe. Which check should they name?
- A solar-output pipe talks to a bureau feed, a store, a monitor, and the model; a new transform is tried beside the old one on the live stream; nobody is sure which code and which file set are in each place. What three pipeline checks should the tester combine?
- The training pipe for a bus-late model was a messy prototype aimed at keeping rows honest. The night-shift operational pipe must stay up, fast, and maintainable. What should drive test emphasis?
- A sidewalk-heave team fitted on last winter’s shed photos. Spring heave on wet brick is the live job. How should representativeness be defined?
- Before comparing piles, testers of a rec-center no-show model spell out use cases, users, sites, expected mixes, and hard edge days — using desk experts, an older booking system, and a trusted public benchmark — then take a stratified sample so every subgroup is in the baseline. Which representativeness step is that?
- Testers plot the candidate fitting set and a reference operational set side by side (histograms, scatter) and check that feature pairings that matter in the yard still appear. What should they use?
- A recycling pile has a dense clump of clear bottles, no night bags, and almost no multi-material class. What representativeness findings are those?
- A solar-output team wants more than a looks-different comment when they compare the fitting mix with a reference operational mix. Which representativeness step should the tester name?
- A dairy project wants to fit this afternoon and check the pile later if the score is poor. When should representativeness testing be done?
- Once a flood-stage model is live, incoming sensor mixes start to wander from the fitting year. What should the tester continue, without teaching a Chapter 6 method?
- A grain-silo fitting set has empty moisture cells and a few rows with the moisture attribute missing entirely. Which dataset constraint should the tester apply?
- Public-pool logs include a turbidity of -4 on a scale that is only defined from 0 to 40. Which dataset constraint should the tester apply?
- A bus-stop board expects an integer passenger count and received the word “crowded.” Which dataset constraint should the tester apply?
- A rec-league sportsmanship file lists point awards whose daily total is above the board’s stated cap, and some days sit under the stated floor. Which dataset constraint should the tester apply?
- After dropping null night readings, a water-meter set has fewer non-null rows than the operations rule requires for a valid day. Which dataset constraint should the tester apply?
- A water-utility dump contains two identical meter rows and a third that differs by a single transposed digit the rule treats as a near duplicate. Policy allows zero. Which dataset constraint should the tester apply?
- A permit-triage table includes a ticket-number column in which every value is unique, like a raw identifier. Which dataset constraint should the tester apply?
- A tool-library log shows a return timestamp earlier than the checkout, hours-open that do not move with recorded energy use, and one temperature no sensor in the shed has ever produced. Which dataset constraints should the tester apply, and where do they run?
- A supervised orchard-pest identifier was fitted on photos whose tags disagree with the agronomist. Why does label quality matter?
- A trained archivist reviews a sample of handwriting tags using the desk’s labelling guide. Which label-correctness approach is that?
- The same wildlife clip is tagged independently by two people and compared. Agreement is scored (percentage or a chance-corrected coefficient). A low score points to a weak guide, an ambiguous clip, or poor tagging — not automatically to a bad model. Which method is that?
- A recycling project cannot re-tag everything. Testers send near-boundary bags and the contamination class that would close the line to a second pass. How should the review budget be spent?
- A second depot’s late-bus set has a well-trusted tag mix. The new depot’s mix is oddly short on weather-hold. Which label-correctness approach flags that?
- Audio event tags on rink-plant recordings sometimes have a start later than the end, or a span that leaves the clip. Which label-correctness approach should the tester pick?
- During fitting, some pest photos keep a high loss against their tags; a later pass also marks low-confidence items. How should those signals be used?
Model Testing for Machine Learning Systems · 67 questions
- A community blood-bank inventory model keeps under-ordering for one neighborhood clinic. A tester is handed the Chapter 6 risk table. Which named mitigation should come first?
- A community-loom pattern ranker starts promoting designs that copy a living artisan’s signature motif. The risk is an unethical model, not a math miss. Which table mitigation should the tester pick?
- A municipal leaf-collection router is scored on “bags collected.” Crews start looping the same easy blocks. Which table mitigation matches a model that exhibits reward hacking?
- A greenhouse-vent model keeps humidity inside the stated band but slams louvers so often the motors fail. Accuracy still looks fine. Which table mitigation should the tester pick?
- A cider-press yield model has no affordable gold gallons figure for each run. The table lists the test-oracle problem. Which named mitigations should the tester pick?
- A tide-pool species ID model fails when a plate is only slightly scuffed. The risk is lack of AI robustness to unexpected inputs. Which table mitigations should the tester pick?
- A maple-sap yield classifier misses the agreed recall on an independent pile. Which table mitigation matches failure to achieve required model performance measures?
- A snow-fence placement brief says “keep drifts off the road” and nothing else. The table lists poor system requirements. Which mitigation should the tester pick first?
- A bike-share rebalancing model ships with a one-line note and no intended use, measured score, or interface. Which table mitigation matches poor model documentation?
- A clock-tower chime predictor was swapped overnight. Four desks report four different pains: the kiosk dies on first request; the live hit-rate is worse; the new model disagrees with last week’s on the same rows; last winter’s accepted layouts now fail. How should those four update risks be matched?
- A municipal grit-bin refill team is told to “write some docs.” Two named frameworks sit in the syllabus: a concise model overview (intended uses, evaluation, ethics) and a standardized dataset description (motivation, composition, collection, uses). Which review artifacts should the tester name?
- A cheese-cave humidity model is machine-fit, opaque, data-dependent, and swapped every thaw. A regulator asks why testers review a card instead of “just reading the code.” What should the tester explain?
- A bandstand crowd-noise card lists two different intended users, an old accuracy, and no hardware note. What should a documentation review hunt for?
- A wind-turbine icing card describes the algorithm and the training pile but is silent on whether the test pile was independent and whether adversarial or functional activities were run. Which documentation checklist section should the tester flag?
- A fountain algae model’s card never states the deploy environment, monitoring alerts, retraining or rollback plan, or how the model will be retired. Why are those contents in-scope for the review?
- A community apiary winter-loss model is treated as high-risk by the city’s AI rule. Go-live is blocked until a documentation audit passes. What extra purpose does that review serve?
- A ferry boarding-wait model is right on the first Friday boat. A sponsor wants a pass sticker. What should the tester require instead?
- An avalanche-beacon “burial likely” model’s brief only says “be accurate.” Testers cannot size the pile. What must acceptance criteria state?
- A library hold-time model looks strong on the slice used to nudge hyperparameters. Which pile may be used for the statistical ML functional-performance test?
- A beehive mite-risk model is scored on last spring’s calm yards. Summer yards are hotter and louder. What must be true of the test pile?
- A compost-odor model’s run finishes. A slide says “87% and we ship.” How should the tester rewrite the report?
- A choir-attendance model is being scored against a stated margin and confidence. After a large prefix of the planned pile, the running score is stably inside the band. What official idea may testers use, without cherry-picking a lucky streak?
- A street-tree failure model sits on a path over a playground. The safety desk wants a stricter reliability claim than a typical accuracy band. What kind of statistical criterion may testers apply?
- A parking-occupancy team can afford a bigger test pile. One camp wants to keep the confidence level and shrink the margin. The other wants to keep the margin and raise the confidence. What do those two official trade-offs mean?
- An ice-fishing hut-permit photo model is shown a plate that looks unchanged to the clerk but is now scored as the wrong hut class. What should the tester name that successful input and the activity that found it?
- A mushroom-cap ID kiosk fails on a naturally scuffed cap and, separately, a researcher asks whether someone could aim such a change at the model. What distinction should the tester keep?
- A hydrant-flow camera model is about to go live. A tester argues for an adversarial pass. What is the official purpose of that pass?
- Testers cannot see inside a pottery-glaze photo model. They raise a similar model whose internals they do know, search that stand-in, then replay the interesting inputs on the original, assuming shared decision boundaries. What approach is that?
- A swimming-lane occupancy camera is a closed box. The desk has no stand-in model. What brute-force black-box approach should the tester recognize?
- A salt-barn bag-count model’s architecture, parameters, and training story are available to the test desk. What should white-box adversarial testing mean here?
- A telescope cloud-cover desk can either sit with a few carefully chosen sky plates or run a tool that produces a large number of variations. How should those two ways of performing adversarial testing be contrasted?
- An oyster-reef cover model has one survey photo that already passed. Testers will change that source using a stated relation and check how the output must move. What technique is that?
- A drawbridge gap-time model is opaque and too costly to simulate by hand. Relational properties would still give confidence. Why should testers choose metamorphic testing rather than a traditional expected-result oracle?
- A playground surface-wear model scored a source photo as “resurface soon,” and that case passed. Testers take a second photo of the same panel under the same light a minute later. What follow-up case and consistency expectation should they derive?
- An alpaca-fleece grader outputs a micron score. A source case passed. Holding other fields fixed, a higher grease-content reading should not produce a finer (lower-micron) grade. What follow-up should testers derive?
- A storm-drain clog model passed on a grate photo. Testers rotate that photo by a small amount that does not hide the debris. What follow-up and invariance expectation should they derive?
- A carousel-load model passed on a quiet Tuesday car. One relation adds an empty duplicate sensor row. Another adds a documented extra rider mass. What should testers derive for those two follow-ups?
- A stained-glass color-match model has never produced a passing source case for a pane family. A writer still wants follow-ups. What limitation should the tester recall?
- A pier-piling tilt model has a passed slack-tide reading. Physics of the berth: a documented higher water level, other fields fixed, should not reduce predicted tilt. How should testers derive the follow-up?
- A sled-hill packing-density desk invented three metamorphic relations over lunch. How should those relations themselves be validated before they mint follow-up cases?
- A kite-festival wind model passes every follow-up from one incomplete relation and still misses an absolute speed error. What should the tester treat that as?
- A skating-pond thickness model was fit on mid-winter sensor rows. Late-season rows are warmer and noisier. What should the tester name that change?
- A CSA-box “will they skip this week?” model still sees the same signup fields, but after a new pickup rule the old “loyal” pattern now predicts the wrong skip. What should the tester name that change?
- After a plow-contract change, a grit-bin model sees new truck IDs in the feed. After a new ice-treatment chemical, the same temperature-and-traffic row should now be called “treat” instead of “hold.” How should those two changes be labeled?
- A dog-park peak-occupancy model is live. Rangers file a current head-count. Testers compare that feedback to the model’s output and to a threshold. What activity is that?
- A marionette-show length advisor asks patrons to rate the suggested runtime after the curtain. How should testers treat that rating in dynamic drift testing?
- A community-garden watering model gets no thumbs-up. Testers instead watch whether beds the model left dry actually dried out. How should testers treat that observed outcome?
- A trolley-bunching model is live, and nobody is labeling every stop. Testers compare the statistical shape of incoming features and of predicted outputs with the shapes seen at fit time, using a named distribution test. What activity is that?
- A darkroom exposure advisor can memorize the training strips, miss the pattern on every strip, or land in between. What three outcomes should testers name, and when does the detection work sit?
- A cemetery plot-demand model recites last decade’s odd memorial-week spikes and then fails on next month’s unseen weeks. What should testers name that behavior?
- A loom-thread tension model looks strong on the validation slice. Testers run an independent test pile that includes uncommon weave patterns the fit never saw. The test score is much worse than validation. What should that gap signal?
- A glass-anneal schedule model is a shallow tree, and the training pile never recorded kiln-zone, which the glass shop says drives the outcome. Both training and validation scores stay poor. What should testers name that?
- A sled-dog rest-stop model posts consistently low accuracy, precision, recall, or F1 on both the training set and the validation set. How should testers read that pair?
- A lighthouse foghorn-timing desk plots learning curves. Training error and validation error stay high and close, with little improvement as epochs pass. What does that picture indicate?
- An orchard-blight model is still in generation. Scores are poor on the fit pile and the tune pile. A colleague calls it “drift.” Another wants a Chapter 3 speech about a thin three-way split. How should testers keep this item?
- A boardwalk crowd model has variant A (current) and variant B (a candidate). Both see the same style of gate counts. Testers will need several runs and a statistical comparison. What approach decides which variant is better?
- A quilt-block ranking desk has no gold “best layout” for every block. They already have last season’s ranker. How should testers treat A/B testing here?
- A garden-watering model was retuned. Agreed ML functional-performance metrics exist. What should A/B testing check before the update replaces the previous variant?
- A trolley-bunching advisor changes itself overnight. Automated checks compare characteristics after the change with those before. When should testers accept the change?
- A dog-park occupancy desk wants a fair call between two models. Which statistical comparisons does the syllabus name for A/B testing of ML systems?
- A marionette-lighting advisor has two implementations. One camp wants to know which scores better on agreed metrics. The other wants a pseudo-oracle that hunts disagreements as defects. How should those jobs be assigned?
- A grit-bin refill desk has no gold “hours-to-empty” for every bin. They run the candidate model and an alternative version on the same inputs and compare outputs. What should testers name that activity and the alternative?
- A cheese-cave humidity pair was built by the same desk on the same open toolkit. Both miss the same midnight spike and “agree.” What must be true of the pseudo-oracle?
- A drawbridge opening model is new. The harbor still has a conventional coded timetable that solves the same “may we open?” question. What may testers use as a back-to-back pseudo-oracle?
- A wind-turbine icing model must answer in a tight field window. Testers raise a slower lab twin that only has to match functional outputs. What is enough for that back-to-back check?
- A lighthouse visibility model is being moved from the development closet to the pier kiosk. Testers feed both environments the same frames. Which check should they pick?
- An orchard-blight pair is scored two ways. A/B would ask which variant wins on F1. Testers instead want disagreements on odd leaf edges as defect clues. Which technique should they pick?
Machine Learning Development Testing · 15 questions
- An apiary winter-loss desk uses a popular ML development framework (TensorFlow or PyTorch named only as examples). A developer called an API in a way the library did not intend, and the fit looks “done.” What mitigation should testers pick?
- A library hold-time team picked a speech-oriented framework to rank tabular hold queues. Fits are clumsy and exports are wrong. What mitigation should testers pick?
- An avalanche-beacon desktop toolkit reports “ready,” but the build skipped a dependency and the first notebook cell dies. What mitigation should testers pick?
- A ferry-wait framework prints a glowing F1 that nobody can reproduce by hand on the same cells. What mitigations should testers pick for that defective evaluation implementation?
- A compost-odor lab’s framework takes minutes to return a one-row score, the screens confuse the seasonal crew, a shared notebook is world-readable, and the user guide is a stub. How should testers match those four framework risks?
- A street-tree failure desk suspects a defect inside a library the framework ships (the syllabus’s example is a defect in PyTorch). Two models that should agree do not, and the official metrics look impossible. Which mitigations should testers pick?
- A parking-occupancy lab used a clustering algorithm on labeled occupancy classes, guessed a learning rate, spilled the same rows into train and test, and called a two-fold split “k-fold.” Which named reviews and tests mitigate those four risks?
- A choir-attendance run looks brilliant on one random seed and collapses on the next; a packed file for the lobby kiosk was never re-scored; a new model is about to replace one that may already be better. How should testers map those three risks?
- An ice-fishing hut-permit kiosk must accept the MLS, pick up GPU-driver and framework dependencies, run the install script, and later uninstall cleanly in the shack and in the harbormaster closet. What should testers name that activity?
- A mushroom-ID handheld is one bad push away from stranding foragers. Testers want to know the previous model (or the whole pipe-plus-model) can be restored. When must rollback testing be performed?
- A hydrant-flow model is ready for the city. Testers will send only a small subset of live requests to the new model and watch latency, accuracy, and error rate before a full rollout. What should they name that activity?
- A pottery-glaze camera already serves the studio. Testers want the new model to see the same live frames without changing what the kiln board is told. What should they name that activity, and how does it differ from a canary?
- A swimming-center occupancy model is converted from its training format into a smaller file for the edge box. What should testers name the check after that format change?
- A salt-barn bag-count MLS is supposed to run on a clerk’s phone, an aisle edge box, and the town-hall server. What should testers name the check across that intended range?
- A telescope cloud-alert MLS publishes an interface that the night-shift feed, a volunteer app, and the data pipe must call. Testers need to check inputs, outputs, error messages, and those integration paths. What should they name that activity?
These questions are original practice material and are NOT actual exam questions or brain-dump content. All vendor marks are trademarks of their respective owners. This site is not affiliated with, endorsed by, or sponsored by the exam vendor.