Evaluation and Monitoring

DB-GENAI · 24 questions

• Meridian Line's team wants to know whether the vector index for the charter-desk assistant is actually surfacing the clause chunk that answers each test question, separate from whether the final generated answer is worded well. Which kind of metric answers that specific question?
• Meridian Line's charter-desk agent performs a retrieval call, then a tool call to look up a counterparty's contract status, then a final LLM call to compose the answer. When a broker reports a wrong answer, the developer wants to see exactly what happened at each of those three steps for that specific request. What MLflow capability provides this?
• During evaluation, Meridian Line's maintenance-triage agent responds to a query about a crane's hydraulic pump with a specific part number that does not appear anywhere in the maintenance logs the retrieval step returned for that query. What has occurred, and what evaluation approach would have caught it?
• Meridian Line's charter-desk clause assistant has a small set of test questions with hand-written reference answers, but the team also wants to evaluate hundreds of real broker queries logged in production, none of which have a reference answer written for them. How should the unreferenced queries be evaluated?
• Meridian Line's port-operations copilot endpoint is serving far more expensively than expected, and profiling shows the chain issues three separate Vector Search retrieval calls per user query, most returning overlapping results, before a single LLM call composes the answer. What is the most direct way to reduce cost here?
• Meridian Line's customs-compliance agent has a curated set of past queries with hand-verified correct HS-code answers. The team wants to compare two candidate prompt versions on this set and see, for each version, the fraction of answers matching the verified correct code exactly. Which evaluation approach fits this specific comparison?
• Meridian Line's platform team wants every request and response served by the port-operations copilot's Model Serving endpoint to be automatically captured for later quality analysis, without adding custom logging code inside the application itself. What Model Serving feature provides this?
• Meridian Line's charter-desk assistant is supposed to answer only from retrieved charterparty clauses and cite the clause it used. During evaluation, several responses give an answer that sounds correct and matches what the reference answer says, but the response includes no clause citation at all and the retrieved context that turn contained nothing relevant. What does this specific pattern indicate, and what should the evaluation flag?
• The safety-review summarizer condenses incident narratives into a one-paragraph summary for the safety officer. The ML team wants a metric that specifically checks whether every claim in the generated summary is supported by content in the source incident report, rather than checking overall summary quality. Which class of metric are they describing?
• Meridian Line's maintenance-triage agent has been running in production for months. The platform team wants to detect, without waiting for a broker or engineer to complain, that the mix of equipment types being asked about has started shifting toward a crane model the retrieval corpus has very little documentation for. Where should this be detected?
• Meridian Line's ML team wants to evaluate the retrieval component of its charter-desk clause assistant in isolation, before any language model touches the results, specifically whether the Vector Search index returns the charterparty clause chunks that actually contain the answer within the top results shown to the generator. Which metric class should they use?
• Meridian Line's port-operations copilot endpoint needs to stay responsive during a sudden surge in queries when a major storm disrupts several ports at once, but the team also does not want to pay for reserved capacity that sits idle the other 350 days a year. Which Model Serving consideration most directly addresses this trade-off?
• Before deploying the customs-compliance agent, Nautilus Customs Brokerage's team needs a labeled evaluation set for its HS-code lookup tool calls, but writing a full hand-crafted reference answer for every possible customs question is too slow to keep pace with development. Which approach lets the team build a workable evaluation set fastest while still producing a meaningful quality signal?
• The charter-desk clause assistant cites a specific clause number in its answer, but a reviewer discovers the cited clause number does not exist anywhere in the charterparty text that was actually retrieved for that query. What class of failure is this, and how should the evaluation pipeline be built to catch it going forward?
• The port-operations copilot answers open-ended planning questions from terminal planners, such as recommending which berth assignment best balances vessel arrival windows, questions with no single correct reference answer to compare against. How should the evaluation team score response quality for these questions?
• The maintenance-triage agent team is iterating on several prompt versions for classifying crane maintenance log severity, and wants to compare each version's evaluation scores, exact prompt text, and model version side by side without maintaining a separate spreadsheet. What should they use?
• Terminal planners report that the port-operations copilot is slow to respond during the morning shift-change rush, and the ops team wants to confirm whether the Model Serving endpoint itself is the bottleneck before investigating the application code. Which built-in signal should they check first?
• Once the safety-review summarizer's Model Serving endpoint is live, the governance team wants a durable, queryable record of every request and response the endpoint has served, so they can later audit for PII leakage or unexpected output patterns. What Databricks feature should the endpoint be configured to use?
• Meridian Line's finance team notices the charter-desk assistant's monthly Foundation Model API bill has spiked, and wants a per-user, per-application breakdown of token consumption to find the cause without adding custom logging code to the application. What should they enable on the serving endpoint?
• The customs-compliance agent occasionally returns a wrong HS code, and the team needs to see exactly which retrieval call, tool call, and intermediate LLM response led to the wrong output for one specific failing example. Which MLflow capability should they use to inspect that?
• The customs-compliance agent calls multiple tools in sequence, an HS-code lookup, a tariff-rate lookup, and a final summarization step, and the team wants an evaluation approach purpose-built for scoring this kind of multi-step, tool-calling agent rather than a single-turn question-and-answer model. Which Databricks capability is designed for this?
• When the customs-compliance agent gives an incorrect answer, the evaluation team wants their harness to tell them whether the failure happened because Vector Search returned the wrong tariff clause, or because the LLM had the right clause in front of it but summarized it incorrectly. How should the evaluation pipeline be designed to make this distinction?
• The maintenance-triage agent now classifies every crane log entry in near real time as terminals file them, producing a steady, high, predictable volume of requests around the clock. Which Foundation Model APIs deployment mode best fits this workload compared to the pay-per-token default?
• Several weeks after a new port-authority circular takes effect, the port-operations copilot's monitoring dashboard shows a rising share of user questions returning low-relevance retrieval results, even though the endpoint's latency and error rate are unchanged. What is the most likely explanation the team should investigate first?