Meridian's multilingual crew reports arrive in English, Tagalog, and Greek, and the team plans to store a language metadata column alongside each chunk so the copilot can later filter or route by language. At what point in the pipeline should the language of each report be determined?
Select an answer to reveal the explanation.
Short Explanation
If you want to filter or route by language later, that field has to exist in the table before the index is built - figuring it out during data prep, while you're already touching every document, is the natural place to do it.
Full Explanation
Determining language during data preparation, before or during chunking, is correct because the stated goal is to have a language metadata column available on each chunk when it is written to the Delta table, and metadata fields must be captured during preparation to be present in the table the index syncs from - this is the same pattern as capturing port, effective-date, or other metadata columns discussed elsewhere in this domain. Determining language after the index has already been built as a manual correction step is inefficient and backwards - it requires re-processing already-indexed content and manually patching metadata rather than capturing it once, correctly, during the pipeline stage designed for exactly that purpose. Assuming the embedding model will silently know the language and expose it as queryable metadata is wrong because an embedding model produces a vector representation of meaning - it does not automatically populate a separate, structured, filterable metadata column in the Delta table; any such column has to be explicitly written during preparation. Asking the end user to specify their query language at query time is wrong because it addresses a different problem - the language of the user's question - not the language metadata needed on the underlying documents themselves, and it still leaves the corpus without the field needed for filtering or routing by source-document language. The exam-relevant point is that metadata like language, like port or date, belongs in the preparation stage so it exists in the table before the index is ever built.