4.0 Configure Evaluation and Monitoring

IBM Certified watsonx Governance Lifecycle Advisor v1 - Associate · 59 questions

• Which task type in watsonx.governance Evaluation Studio is specifically designed to assess retrieval-augmented generation pipelines?
• Which three metric dimensions does watsonx.governance Evaluation Studio provide for LLM evaluation?
• In watsonx.governance LLM-as-a-Judge evaluation, what is the valid score range for metrics such as Faithfulness and Answer Relevance?
• In watsonx.governance LLM-as-a-Judge evaluation, which metric tracks judge LLM calls that fail to return a valid numeric score?
• Which feature in watsonx.governance Evaluation Studio allows a user to designate one prompt or LLM configuration as a fixed baseline during multi-asset comparisons?
• When using multi-asset testing in watsonx.governance Evaluation Studio, what is the primary purpose of assigning weighted factors to evaluation metrics during custom ranking?
• In watsonx.governance, which text quality metric measures n-gram overlap between generated text and reference texts with an emphasis on recall?
• Which text quality metric specifically evaluates text simplification by comparing model output against both a reference text and the original source input?
• In watsonx.governance text evaluation, which metric is defined as the size of the intersection of two token sets divided by the size of their union?
• A data scientist needs a metric that scores 1 when model output is character-for-character identical to the reference and 0 otherwise. Which metric should they configure in watsonx.governance?
• In watsonx.governance, the HAP (Hate, Abuse, and Profanity) index belongs to which evaluation category and monitors which data streams?
• Which statement about PII detection in watsonx.governance is correct?
• Which quality metric is most appropriate when evaluating a binary classification model where false negatives and false positives carry equal business cost?
• In watsonx.governance fairness monitoring, what does the disparate impact ratio measure?
• When configuring a fairness monitor in watsonx.governance, what is the correct definition of the reference group?
• A deployed fraud detection model's input feature distributions have shifted significantly relative to training data, but labeled feedback data shows quality metrics remain within acceptable thresholds. Which type of drift does this scenario describe?
• Which statement best distinguishes LIME from SHAP in the context of watsonx.governance explainability?
• In watsonx.governance explainability, what is a contrastive explanation?
• When configuring a drift evaluation in watsonx.governance, what must be supplied as the reference before you can set the drift alert threshold?
• In the watsonx.governance monitoring configuration workflow, which action must be completed before configuring drift detection?
• A model risk manager wants watsonx.governance to automatically initiate a reassessment workflow whenever a fairness metric remains below 0.80 for three consecutive evaluation windows. Which capability supports this requirement?
• How does watsonx.governance 2.0 differentiate its monitoring approach from earlier-generation model risk management platforms?
• Which watsonx.governance artifact is automatically generated to provide a standardized summary of a model's intended use, training data, performance metrics, and known limitations for stakeholder and regulatory review?
• A monitoring dashboard shows that AUC-ROC for a binary classifier has remained above 0.85 but statistical parity difference has deteriorated to -0.22 over the past 30 days. What does this combination of signals indicate?
• A lending-eligibility model is due for production rollout, and the sign-off criteria must catch a fairness gap before it reaches customers. What should the evaluation plan include?
• No process exists for detecting when a deployed model's behaviour starts to change. What should the monitoring plan add to close that gap?
• A customer wants a scalable way to evaluate its generative AI outputs. What should that evaluation cover?
• A governed production model needs risk caught before it causes real damage. What should be configured to make that happen?
• A pilot deployment's model reports 92% overall accuracy, yet a stakeholder still flags a fairness concern. What is the right next step?
• The risk that a model's real-world performance quietly degrades weeks after launch needs mitigating. Which choice best addresses it?
• Shipping an ungrounded assistant into production is a real risk. Why is checking that generated answers are grounded in a reliable source particularly important for enterprise use?
• A senior engineer reviews the monitoring plan and notices that alerts are only set to fire after a model's error rate has already caused visible customer complaints. What is wrong with this design?
• A customer building a scalable image-classification service argues that because the training data was anonymised, fairness testing can be skipped. How should the consultant respond?
• A proof of concept is being turned into a governed production service. Which monitoring capability should be added that the proof of concept did not need?
• A pilot chatbot's evaluation only measures how long its answers are. What does this evaluation approach fail to catch?
• Model drift needs to be caught before it affects customers. Which monitoring configuration best supports that goal?
• Ahead of a production rollout, the sign-off checklist must rule out shipping a model with an undetected fairness gap. Which evaluation step directly targets that risk?
• A senior engineer notices the current plan proposes disabling drift alerts because they generate too much noise. What is the risk of that decision?
• A customer building a scalable implementation is considering letting end users flag unsafe generative AI outputs as the only safety check. Why is this insufficient on its own?
• Someone must decide who receives alerts when a threshold is crossed for a newly governed model. What is the correct approach?
• During a pilot, complaint volume from one demographic segment is noticeably higher than from others, even though the model's overall accuracy metric looks healthy. What does this pattern suggest an administrator should do?
• Monitoring only application server CPU comes up as insufficient for a deployed model. What is missing from that approach?
• A generative assistant is heading toward a production rollout. Which task should be included in its pre-launch evaluation plan?
• A senior engineer reviews the current monitoring plan and finds that thresholds were copied unchanged from an unrelated model in a different domain. What risk does this create?
• A customer's evaluation results show a performance gap between two demographic subgroups on an otherwise strong-performing model. What does this finding mean for the rollout?
• A project lead assumes that because a model performed well during validation, it will keep performing well indefinitely once deployed. Why is that assumption risky?
• A pilot assistant occasionally produces confident-sounding but false answers. Which evaluation criterion targets this specific symptom?
• Thresholds and alerts need a place in the broader governance lifecycle, beyond just technical monitoring. What is the best answer for how they fit?
• Two systems are queued for production rollout: an internal document-search tool and a customer-facing insurance-pricing model. Which one most clearly requires group-level fairness evaluation, and why?
• Which task is drift monitoring specifically designed to accomplish?
• A consultant recommends evaluating a generative assistant for groundedness, relevance, harmful content and policy compliance. Which of these should be applied to every generative AI use case regardless of domain?
• A governed production model needs a decision on how often its thresholds should be reassessed after launch. What is the best practice?
• A pilot has been running in production for several months. Which practice keeps fairness evaluation meaningful over that time rather than just at launch?
• A team wants to know as soon as a deployed model's behaviour meaningfully changes. Which practice gives them that visibility?
• Why should generative AI evaluation include a check for harmful content in addition to groundedness and relevance?
• Which statement best summarises the purpose of configuring thresholds and alerts in AI governance?
• Which statement best summarises how fairness should be evaluated for a deployed AI system?
• Summing up, why must a governed production deployment include drift monitoring even though the model already passed validation?
• Which approach best summarises how a generative AI system's outputs should be evaluated before wide release?