A scanned bill of lading contains a templated grid of fixed fields (shipper, consignee, port of loading, container number) plus a short free-text 'remarks' section. For indexing purposes, how should a data engineer treat these two parts?
Select an answer to reveal the explanation.
Short Explanation
A bill of lading is really two different kinds of information glued together - a form with fixed slots and a few lines someone actually wrote - so split them: the form fields become searchable metadata (filter by container number or port), and the handwritten-style remarks are what actually deserves a semantic embedding.
Full Explanation
Extracting the templated fields as structured metadata while chunking the free-text remarks separately is correct because a bill of lading mixes two fundamentally different content types: fixed-format fields best suited to exact-match or filter-style metadata search, and short free-text commentary best suited to semantic embedding. Treating the whole page as one undifferentiated chunk is wrong because it forces a semantic embedding to represent structured key-value data it was never designed to encode well, and it prevents precise metadata filtering on fields like container number or port of loading later at query time. Discarding the templated fields entirely is wrong because those fields - shipper, consignee, port, container number - are exactly the kind of metadata that should drive filtering in Vector Search queries; throwing them away removes a valuable retrieval lever. Discarding the remarks as noise is backwards - free-text remarks are frequently where the substantive, non-boilerplate information lives (an exception noted at loading, a discrepancy flagged by the crew), so treating them as noise while keeping only the fixed fields inverts which part is actually the noise and which is the signal. The exam-relevant distinction is separating structured metadata extraction from semantic chunking rather than treating a mixed-format document as uniform text.