Data Preparation

DB-GENAI · 28 questions

• Meridian's charter-desk assistant needs to retrieve the exact clause a broker is asking about from a 40-page charterparty contract, without pulling in half of an unrelated clause. Which chunking approach best fits this source document?
• Port-authority circulars at Meridian's terminal follow a consistent structure: a header block, a 'Notice Type' field, a dated body section, and a 'Superseded By' footer. Which chunking approach best preserves retrieval quality for this source?
• Crane-maintenance log entries at Meridian's terminal are short, telegraphic lines like 'Crane C-14 hydraulic pressure low, replaced seal, cleared 08:42.' A data engineer is deciding how much character overlap to configure between chunks for this source. What is the most defensible choice?
• A scanned bill of lading contains a templated grid of fixed fields (shipper, consignee, port of loading, container number) plus a short free-text 'remarks' section. For indexing purposes, how should a data engineer treat these two parts?
• Meridian's bills of lading arrive as scanned image PDFs with no embedded text layer. Which approach should a data engineer use to get usable text into the preparation pipeline?
• Every customs declaration PDF that Meridian's brokers file includes an identical two-paragraph legal disclaimer on page one, repeated verbatim across thousands of filings. Left in the corpus, what is the main risk to the retrieval index?
• A GenAI engineer notices the charter-desk assistant sometimes answers demurrage questions using text from a laytime clause instead, and sometimes truncates a valid answer mid-sentence. The chunk size is currently very small (roughly 100 characters). What is the most likely fix?
• Meridian's port-operations copilot corpus contains a circular that has been re-issued four times over two years with only the effective date and one paragraph changed each time; the rest of the notice is boilerplate carried forward unchanged. What is the recommended data-preparation step?
• Port-authority circulars are published as HTML pages on the authority's public notice board, complete with navigation menus, a sidebar of unrelated links, and the actual notice text in the page body. What is the most appropriate extraction approach?
• Meridian wants its charter-clause Delta table to stay continuously synchronized with a Delta Sync Vector Search index, so that a newly added clause chunk appears in the index without a manual full re-index. What must be enabled on the source Delta table for this?
• After extracting and chunking Meridian's customs declarations, where should the resulting text chunks and their metadata land before a Vector Search index is built over them?
• Meridian's charterparty contracts are issued as digitally generated PDFs (not scans) with a normal embedded text layer. Which extraction approach is most appropriate and efficient?
• Before building the customs-compliance agent's retrieval index, a data engineer audits the corpus and discovers that HS-code tariff schedules for two of the five jurisdictions Meridian operates in are missing entirely. What data-preparation step does this call for?
• Meridian's data engineer is setting up a Vector Search index over the port-circular corpus and must decide whether Databricks computes the embeddings automatically from the Delta table's text column, or whether the team pre-computes embeddings themselves and stores them as a vector column. What does this decision primarily determine?
• Meridian's engine-room crew files maintenance and safety reports in English, Tagalog, and Greek depending on the crew member on duty. The port-operations team wants one unified retrieval index across all three languages. What should most directly drive the embedding-model choice?
• A data engineer is choosing between two embedding models for the customs-compliance corpus: Model X produces 384-dimension vectors at lower cost, Model Y produces 1536-dimension vectors at higher cost and generally stronger semantic quality. What is the correct way to frame this decision?
• Safety incident narratives at Meridian's terminal are free-text reports of varying length, from a two-sentence near-miss note to a two-page injury investigation, and they contain crew members' names and other PII. Which chunking approach is most appropriate for this source?
• An audit of the port-circular corpus finds that 30% of indexed notices have already been superseded by a newer circular, but the index has no way to distinguish current from superseded notices at query time. What is the most direct data-preparation fix?
• Meridian's multilingual crew reports arrive in English, Tagalog, and Greek, and the team plans to store a language metadata column alongside each chunk so the copilot can later filter or route by language. At what point in the pipeline should the language of each report be determined?
• The port-operations copilot needs to answer questions scoped to a specific port and a specific date range, such as 'what circulars were active at the Rotterdam terminal in March.' Beyond the chunked notice text itself, what must the data-preparation pipeline also produce for this requirement to be satisfiable?
• A customs filing document contains both a dense structured table of HS codes and tariff rates, and several paragraphs of prose explaining exemption conditions. Should these two content types be chunked and prepared the same way?
• Meridian licenses several standard charterparty templates (e.g., a common industry form) and each contract begins with an identical, several-hundred-word 'definitions and interpretation' preamble common to that template. Across hundreds of contracts using the same template, what data-preparation step improves retrieval quality?
• A data engineer is finalizing the chunk-size setting for the charter-contract corpus and checks the embedding model's maximum input length. Why does that maximum matter when choosing a chunk size?
• Some clauses in Meridian's charterparty contracts are long enough that even clause-boundary-aware chunking must further split a single clause into two chunks. In that specific case, what does configuring a modest character overlap between the two resulting chunks accomplish?
• For the safety-review summarizer, a data engineer is choosing between chunking each incident narrative as one large chunk per report (regardless of length) versus splitting long reports into multiple smaller, paragraph-based chunks. A two-page injury investigation report is the case in question. Which approach better supports precise retrieval for a query about one specific detail buried in that report, such as 'what corrective action was recommended'?
• Meridian's crane-maintenance logs are stored as a plain CSV export from the terminal's maintenance-management system, with columns for equipment ID, timestamp, technician, and a free-text notes field. What is the most appropriate extraction approach for this source?
• The maintenance-triage agent needs to be able to answer 'what was the most recent service performed on Crane C-14,' not just 'what services have ever been performed on Crane C-14.' What must the data-preparation pipeline write to the Delta table to make that distinction answerable?
• Before Meridian's data-preparation team declares the charter-desk corpus 'ready' for indexing, what should they verify about document-type coverage, beyond simply confirming that files were successfully extracted and chunked?